acceptodds
Withdrawn from ICLR 2027

Subdivision Measure Matching: Generative Modeling in Integral Form

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generating samples from distributions with fractal, sharp, sparse, or disconnected support requires reproducing fine-scale structure while preserving distinct modes and gaps. While learned quantizers suffer from codebook collapse or depth saturation, increasing the resolution of scalar grid quantizers leads to exponential growth in the prediction head. To address these limitations, we introduce Subdivision Measure Matching (SMM), which models probability mass through codebook-free recursive subdivision of the domain. SMM progressively refines coarse regions into fine samples, increasing precision exponentially with depth while keeping the categorical prediction dimension fixed. For faster sampling, we introduce parallel coordinate refinement at finer levels, requiring only one model evaluation per level. Theoretically, SMM achieves optimal reconstruction accuracy for its code length and codebook size, with a strict worst-case advantage over shared additive residual quantization, and vanishing factorization-error bounds under standard smoothness assumptions. Experiments across 7 toy distributions demonstrate superior recall, coverage, and depth scaling compared to residual quantization baselines. On D3IL robot imitation tasks, SMM outperforms baselines, including diffusion policies, in 8 of 9 settings and achieves higher behavioral diversity on 3 tasks. On high-dimensional image synthesis, SMM improves ImageNet FID from 15.48 to 13.17 over a Flow Matching baseline.

Primary area:
generative models
Keywords:
Vector Quantization, Generative Modeling, Autoregressive Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wESsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Segmentation Measure Matching (SMM). The data are first normalized by marginal CDF to [0,1]^D. Each coordinate is then represented by its cardinality-K numeric path through T-layers subdivided recursively by K-elements. A transformer with a custom interleaved full/coarse labeled attention mask (KP3Attention, Figure 2, p.6) can autoregressively predict the classification distribution over the K^P joint subunits of each patch over patches and depths. Training uses cross-entropy with uniform sampling depth (Equation 4-5, page 6). The paper expresses this as the exact maximum likelihood of a finite-resolution discretization. Temperature, logit space CFG and deeper parallel refinement variants are described in Appendices A.2 and A.9.

Theoretical results appear in Appendices B–H (pp. 21–30). Proposition 1 shows that for a fixed symbol budget, the midpoint grid is minimax optimal in ℓ∞ and strictly outperforms the shared, unscaled, and decoded RQ. Theorem 2 gives the multi-scale W1 bounds for each level of condition KL. Theorem 3 restricts the factorization KL of parallel refinements of smooth positive logarithmic densities. Appendix H gives cut-through gradient difference bounds.

The experiments cover four settings. The first was a seven-toy distribution (Table 1, page 8; Table 4, page 16). The second is the D3IL robot imitation, where SMM reports the best success in the 8/9 setting (Table 2, page 8) and the highest entropy in 3/9 (Table 5, page 17). The third is pixel-space generation on a small 28×28 dataset and CIFAR-10, scored using PCA-FID (Table 3, page 9; Table 6, page 18). The fourth is the ImageNet 256 latent generation, where SMM-B achieves FID 13.17, flow matching 15.48, CFG 8.88, and SMM-XL 4.30 (Table 7-8, pp. 18-19). Ablation covers parallel refinement (Table 12-13, page 21) and reconstruction depth scaling (Figure 6-7, page 22-23).

Critical strengths

The formula is simple and coherent. Representing distributions by conditional cell mass ratios on nested K-ary partitions provides tractable exact likelihoods at finite resolution, log-space temperature and CFG, and fixed K^P head size and accuracy as K^T grows (Section 3, pp. 4-6). Toy evidence supports the exciting claim that this contributes to non-smooth support. On the Sierpinski target, SMM obtained recall 0.998/0.996 and coverage 0.906/0.909, while traffic matching was 0.909/0.920 and 0.670/0.641 (Table 1, page 8). Figure 4 (page 16) shows that at NFE=1000 the flow match is still filling the fractal hole.

The empirical scope is broad and some of the analyzes rich. A deep scaling study (Figure 6-7, pp. 22-23) shows that the per-level error rate of SMM remains around 1/K. RQ instead becomes saturated and its effective code usage drops significantly. Parallel refinement ablation (Table 12, page 21) shows that classification refinement avoids the mean collapse of MSE regression: FID 0.407 vs. 4.491 (with one AR level), with 7.7 times fewer consecutive steps. D3IL results are competitive in all nine settings. An example is Sort-4, where the optimal baseline is 64.4 vs. 56.5 (Table 2, page 8).

Critical weaknesses

Its novelty and theoretical contributions have been overstated relative to its content and related work. Digital or bit-plane extended autoregressive modeling with coarse-to-fine value accuracy, closely related to subscale/bit-depth upscaling autoregressive image models and discrete likelihood models. Section 2.2 (pages 3-4) does not position SMM against these; it only mentions Pólya trees and histograms. Proposition 1 (p.24) is the standard fact that for M points a uniform grid is ℓ∞-minimax. "Strict dominance" only applies to shared, unscaled, and decoded RQs. Theorem 3 (pp. 25-26) requires a strictly positive C^2 log density, which rules out the fractal, sparse, and disconnected goals that drive this paper (Abstract, p. 1).

The comparison of several titles is difficult to explain. Pixel-space results using PCA-FID have a value of 0.163 on CIFAR-10 (Table 3, page 9). These are not comparable to the literature and although there were 3 seeds, no seed differences were reported (A.6, p. 17). On ImageNet, flow matching obtained 15.48 without CFG but 15.269 with CFG 3.0 (Tables 7, 9), indicating insufficient baseline tuning. MAR 19.37 is a reimplementation of unclear configuration. SMM-XL's 4.30 (Table 8) was not compared with LlamaGen, despite sharing its marker recipe and backbone (A.7, page 18). The D3IL baseline was copied from Jia et al. instead of rerunning (p.7), and 6/9 tasks had lower entropy (Table 5).

Questions for authors

How to adjust flow matching baseline (sampler, NFE, CFG schedule, training length) on ImageNet? Why does CFG 3.0 barely change its FID (15.48 vs. 15.269)? How does SMM-B/L/XL compare to the released LlamaGen-B/L/XL given the same tokenizer and training budget? Are the RQ-VAE, VQ-VAE, FSQ-VAE, and MAR rows in Table 7 trained with matching backbones and epochs?

Can you report the standard (initial) FID for CIFAR-10 or other metric comparable to the literature, and the standard deviation for the three seeds? Do the pixel-space conclusions in Tables 3 and 6 hold under these metrics?

How do SMMs conceptually and empirically differ from previous autoregressive models that decompose pixel values ​​into bit planes or coarse-to-fine value numbers? Is it possible to include such baselines, such as PixelCNN or Transformer style models, with 256-way classification per pixel under matching parameters?

For D3IL, were the 6 seeds of the SMM trained and evaluated on the launch count and initial state distribution of the baseline? The very small standard deviation (e.g., 0.2 for "Avoid") contrasts with published baselines. Can you explain why behavioral entropy drops significantly on Pushing (0.612 vs. 0.871) and Stack-1/2?

1 comment
AuthorsOpenReview ↗

Response to Reviewer wESs

We thank the reviewer for their time and valuable feedback. Please find the responses to the raised concerns below:

1. Related work (W1 & Q3)

Thank you for raising this point. We clarify the differences between SMM and related coarse-to-fine and discrete likelihood models below:

  • Subscale Pixel Networks [R1]: SPN is a PixelCNN-based, pixel-by-pixel autoregressive model for images only. It conditions on an encoder of previously generated slices and uses 2 stages with a separate network for each.
  • Infinity [R2]: Infinity is hierarchical over spatial resolution (next-scale prediction), whereas SMM is hierarchical over value precision and makes a joint KPK^P prediction at each level.
  • Fractal Generative Models [R4] apply recursion over spatial patches, and pixels are predicted with a flat 256-way softmax. In contrast, SMM recursively subdivides the value domain.

2. Prop. 1 and comparison with RQ (W1)

We agree that the strict advantage over RQ is established for the worst case. We consider the worst case to establish a theoretical bound. Otherwise, a theoretical bound cannot be established, and the comparison has to be made empirically. The empirical results are available in Fig. 6 and Fig. 7 (Appendix A.10), where SMM's reconstruction error keeps decreasing with depth while RQ's plateaus.

3. Comparison with 256-way classification per pixel (Q3)

A flat softmax head grows exponentially with the number of channels. For example, with our ImageNet configuration (K = 4, T = 2, P = 7), a flat head requires KPT=414≈2.7×108K^{PT} = 4^{14} \approx 2.7 \times 10^8 classes per patch, making it not practically applicable.

Thank you for the suggestion to compare with the coarse-to-fine model. We train SPN [R1] on MNIST and KMNIST and evaluate it under the same setup:

DatasetMethodFID ↓Precision ↑Recall ↑
MNISTFlow Matching0.8070.8360.937
VQ-VAE11.4680.9970.006
FSQ-VAE3.2110.7040.13
RQ-VAE2.820.970.367
SPN0.540.7380.95
Ours0.4220.810.949
KMNISTFlow Matching2.6360.7310.796
VQ-VAE2.3180.3770.94
FSQ-VAE6.2430.950.133
RQ-VAE2.3760.4490.941
SPN1.9040.6520.869
Ours1.6930.6430.81

4. Thm. 3 and fractal targets (W1)

We agree that Thm. 3 does not apply to fractals. It applies to high-dimensional images, where parallel refinement is used (Tab. 12 and Tab. 13).

5. Baselines in Tab. 7 and comparison with LlamaGen (W2 & Q1)

Following LlamaGen [R6], we use the VQ-8 version (however we change latent channel size from 8 to 7). We could not use the default 8-channel latent version due to the exponential explosion of the softmax number of choices. In Tab. 7, VQ-VAE uses LlamaGen setup (7-dim VQ-8 version) and is trained with the same protocol as SMM (50 epochs).

6. D3IL baselines and behavior entropy (W2 & Q4)

We retrain BESO and DDPM-ACT, and add VQ-BeT [R5]. Success rates (%), six-seed means:

MethodAvoidingAligningPushingSort-2Sort-4Sort-6Stack-1Stack-2Stack-3
BESO (retrained)29.882.376.981.231.00.189.069.19.0
DDPM-ACT (retrained)51.985.993.085.816.20.550.221.93.4
VQ-BeT73.373.177.774.713.30.412.30.30.0
SMM97.091.495.489.464.42.893.472.614.4

The retrained baselines are comparable to the published numbers on most tasks. SMM outperforms the retrained baselines and VQ-BeT on all nine tasks.

Regarding the lower behavior entropy on Pushing and Stack-1/2, our policy produces more predictable paths.

[R1] Menick, J., & Kalchbrenner, N. (2019). Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference on Learning Representations.

[R2] Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., & Liu, X. (2025). Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15733-15744).

[R3] Morin, F., & Bengio, Y. (2005). Hierarchical probabilistic neural network language model. In International Workshop on Artificial Intelligence and Statistics (pp. 246-252). PMLR.

[R4] Li, T., Sun, Q., Fan, L., & He, K. (2025). Fractal generative models. Transactions on Machine Learning Research.

[R5] Lee, S., Wang, Y., Etukuru, H., Kim, H. J., Shafiullah, N. M. M., & Pinto, L. (2024). Behavior generation with latent actions. In International Conference on Machine Learning (pp. 26991-27008). PMLR.

[R6] Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., & Yuan, Z. (2024). Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525.

Reviewer_bfhsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Subdivision Measure Matching (SMM), which generates samples by recursively subdividing the data space and predicting each subregion’s probability mass without learned codebooks. It uses cross-entropy training and coarse-to-fine sampling, with optional parallel refinement to accelerate generation. Experiments show improved coverage on toy distributions, stronger robotic imitation performance, and ImageNet FID improving from 15.48 to 13.17 over the evaluated flow-matching baseline.

Critical strengths
  • The paper is well written, with clear coverage of the background and preliminaries that makes the work accessible to readers.
  • The experimental evaluation is comprehensive, covering diverse datasets ranging from MNIST to ImageNet.
Critical weaknesses
  • The practical sampling efficiency is insufficiently established. Beyond the analysis in Section 4.5, all quantitative comparison tables should explicitly report the number of sampling steps for both the proposed method and the baselines. This would help readers assess the trade-off between sample quality and computational cost.
  • The ImageNet results are relatively modest compared with stronger methods reporting FID scores around 2. Improvements over a relatively weak baseline provide limited evidence of the proposed method’s effectiveness. Comparisons with stronger baselines under matched settings, including image resolution and sampling budget, would make the empirical claims more convincing.

To the authors: I used an AI tool to help analyze the manuscript and improve my understanding. I formulated my own human-based concerns and used the tool to polish their wording. I carefully reviewed the revised text to ensure that it accurately reflected my original concerns.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer bfhs

We thank the reviewer for their time and valuable feedback. Please find the responses to the raised concerns below:

1. Sampling efficiency (W1)

Thank you for the suggestion. SMM draws a sample in N × T sequential categorical steps, where N is the number of patches and T is the subdivision depth (Sec. 4.5). The sampling cost therefore scales linearly with the number of patches and logarithmically with the target precision, and joint-patch prediction reduces the number of sequential steps by a factor of P.

For faster sampling, our parallel refinement (Appendix A.9) requires only one model evaluation per level after the first autoregressive levels. As shown in Tab. 12, on MNIST it reduces the sequential steps 7.7 times.

On ImageNet, we report FLOPs per sample with KV caching:

MethodFLOPs per sample ↓
Flow Matching21.3 TFLOPs (42.6 with CFG)
SMM0.55 TFLOPs (1.11 with CFG)

SMM requires about 39× fewer FLOPs per sample than Flow Matching, showing that SMM is more efficient at inference.

Reviewer_J2MMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Subdivision Measure Matching (SMM). SMM first normalizes data to [0,1]D[0,1]^D through marginal CDFs. It then represents each point by its root-to-leaf path in a KK-ary axis-aligned subdivision, and learns that path with a transformer. The model factorizes autoregressively over patches of PP coordinates (Eq. 3) and, within each patch, over depths (Eq. 2). Each step is a KPK^P-way categorical draw. A custom attention mask (KP3Attention) interleaves "complete" and "coarse" tokens. Training uses per-step cross-entropy with a uniformly sampled depth, which the authors describe as exact MLE of the depth-TT discretization (Eq. 5). A parallel-refinement variant samples all coordinates independently, one level at a time, after rr autoregressive levels.

The theory has four parts:

  • Prop. 1: the uniform midpoint grid is ℓ∞\ell_\infty-minimax optimal for a given symbol budget, and it is strictly better than shared, unscaled, sum-decoded RQ.
  • Thm. 2: a multiscale W1W_1 bound in terms of per-level conditional KL.
  • Thm. 3: an O(ht4 Ωoff)O(h_t^4\,\Omega_{\text{off}}) bound on the KL cost of factorizing the refinement across coordinates, assuming a C2C^2 positive log-density.
  • App. H: STE-discrepancy bounds that decay as K−TK^{-T}.

The experiments cover 7 toy distributions (including fractals), D3IL state-based imitation (best success on 8/9 tasks), six 28×28 datasets plus CIFAR-10 in pixel space (PCA-FID), and ImageNet-256 latents. On ImageNet, SMM-B reaches FID 13.17 against 15.48 for a matched Flow Matching baseline, and SMM-XL with CFG reaches 4.30.

Critical strengths
  1. A simple, exact-likelihood formulation that handles irregular supports. Representing a measure by conditional mass ratios on a nested partition, in the spirit of a Pólya tree, and learning those ratios with a shared transformer and a fixed-size head is clean. It gives exact MLE on the discretized law, logit-space temperature, CFG, and GRPO for free, and it needs no smoothness of the target. The Sierpinski and grid results (Fig. 4, Tables 1 and 4) show this concretely: continuous transport blurs the gaps even at 1000 NFE, while SMM resolves them in about 5–10 levels. This is the paper's most convincing evidence that the representation matters.

  2. Strong imitation-learning results plus a well-posed parallel-refinement analysis. D3IL success is best on 8 of 9 tasks, with low variance (for example, Sort-4 rises from 56.5 to 64.4 and Stack-3 from 12.0 to 14.4). This is a setting where sharp, multimodal action distributions matter in practice. Separately, Thm. 3 is a non-trivial and useful result. It shows the KL cost of coordinate-independent refinement decays as ht4h_t^4 at fine levels. This justifies the AR-coarse/parallel-fine split empirically validated in Tables 12–13: a 7.7× reduction in sequential steps with no loss in FID on MNIST.

Critical weaknesses
  1. The novelty relative to digit- and bit-wise autoregression is overstated, and the theoretical "advantages" are largely classical or made against a weak comparator.

    • Once data are discretized to KTK^T levels per coordinate, SMM is an autoregressive model over the base-KK digits of each value. This is a chain-rule refactorization of a flat KTPK^{TP}-way categorical. Closely related prior work is not discussed or compared:
      • Subscale Pixel Networks (Menick & Kalchbrenner, 2019), which use depth (bit) upscaling
      • bitwise AR, as in Infinity (cited only in passing)
      • hierarchical/tree softmax heads
      • Fractal Generative Models (Li et al., 2025)
      • VQ-BeT for robotics
    • Prop. 1 is the standard volumetric covering bound, attained by any uniform grid.
    • The "strict advantage over RQ" holds only for a shared, unscaled, sum-decoded codebook under worst-case error over the whole cube. That metric is not the relevant one for learned, data-adaptive quantizers on concentrated data.
    • The expressivity of SMM equals that of a flat per-patch softmax, so the real claims are about head efficiency and inductive bias. These need to be isolated empirically (see Q1).
  2. The baselines and metrics make the headline image results hard to calibrate, and the main theory does not cover the headline regime.

    • Pixel-space results use PCA-FID, with values such as 0.163 on CIFAR-10 that are not comparable to the literature.
    • On ImageNet, the Flow Matching (15.48), MAR (19.37), and RQ/VQ baselines are all far below their published numbers at similar scale; for example, MAR-B reports roughly 2.3 FID. SMM-XL at 4.30 is also behind LlamaGen-XL, even though it uses the LlamaGen autoencoder recipe and backbone.
    • The most natural baseline, LlamaGen with its own VQ tokenizer and an identical backbone and schedule, is missing.
    • With T=2T=2 and K=4K=4, the ImageNet SMM is close to a two-stage FSQ with 16 levels per channel, so the FSQ-VAE configuration needs to be specified exactly.
    • Thm. 3 assumes a strictly positive C2C^2 log-density. That excludes exactly the fractal, sparse, or disconnected targets used to motivate the method, so the parallel-refinement guarantee does not apply where SMM is argued to shine most.
    • On D3IL, SMM's behavior entropy is markedly lower than the baselines on several tasks (Pushing 0.61 vs. 0.87, Stack-1 0.15 vs. 0.76). This suggests part of the success gain comes from mode concentration, which the text does not discuss.
Questions for authors
  1. Under an identical backbone, data, and schedule, how does SMM compare with (a) a flat KTPK^{TP}-way (e.g., 256-way per pixel for P=1P=1) autoregressive softmax and (b) a bit/digit-plane AR such as SPN-style depth upscaling? This is the key ablation for attributing gains to the subdivision factorization rather than to the transformer and the training recipe.
  2. ImageNet:
    • Is the encoder trained through the SMM quantizer with STE (as App. H suggests), or is SMM fit post hoc to a frozen continuous autoencoder?
    • What are the exact FSQ-VAE levels and training budget?
    • What do you get with LlamaGen-B and its own VQ under your 50-epoch schedule, and what is the wall-clock sampling time against Flow Matching, given N×T=2048N \times T = 2048 sequential steps?
  3. Why report PCA-FID rather than standard Inception FID on CIFAR-10? Please provide standard FID for CIFAR-10 at least.
  4. D3IL: Were the baselines retrained under your 500-epoch protocol, or taken from Jia et al.? If the latter, can you report at least BESO and DDPM-ACT retrained under your setup, plus VQ-BeT?
  5. How sensitive are the results to the marginal-CDF normalization? For multimodal marginals, the CDF warps cell geometry, so the uniform-grid optimality in Prop. 1 then holds in CDF space rather than in data space.
  6. The objective used in A.1 (random terminal depth RR, complete tokens at depth RR) is not the path NLL of Eq. 5. Does the claim of "exact MLE" still hold for the trained model?

Suggestions (future work, not affecting my score):

  • Extend Thm. 3 to measures without densities, for example via self-similar measures or a doubling-measure condition.
  • Explore non-axis-aligned or learned partitions (already noted in the Limitations section).
2 comments
AuthorsOpenReview ↗

Response to Reviewer J2MM (1/2)

We thank the reviewer for their time and valuable feedback. Please find the responses to the raised concerns below:

1. Related work (W1)

Thank you for pointing out these references. We clarify the differences with each of them below:

  • Subscale Pixel Networks [R1]: SPN is a PixelCNN-based, pixel-by-pixel autoregressive model for images only. It conditions on an encoder of previously generated slices and uses 2 stages with a separate network for each.
  • Infinity [R2]: Infinity is hierarchical over spatial resolution (next-scale prediction), whereas SMM is hierarchical over value precision and makes a joint KPK^P prediction at each level.
  • Tree softmax [R3] is a language-modeling method, while SMM subdivides a continuous data domain.
  • Fractal Generative Models [R4] apply recursion over spatial patches, and pixels are predicted with a flat 256-way softmax. In contrast, SMM recursively subdivides the value domain.
  • VQ-BeT is a VQ-based method with a learned codebook, whereas SMM has no codebook. We also add VQ-BeT as a baseline on D3IL (see 9).

2. Comparison with RQ in Prop. 1 (W1)

We agree that the strict advantage over RQ is established for the worst case. We consider the worst case to establish a theoretical bound. Otherwise, a theoretical bound cannot be established, and the comparison has to be made empirically. The empirical results are available in Fig. 6 and Fig. 7 (Appendix A.10), where SMM's reconstruction error keeps decreasing with depth while RQ's plateaus.

3. Comparison with a flat softmax (W1 & Q1a)

A flat softmax head grows exponentially with the number of channels. For example, with our ImageNet configuration (K = 4, T = 2, P = 7), a flat head requires KPT=414≈2.7×108K^{PT} = 4^{14} \approx 2.7 \times 10^8 classes per patch, making it not practically applicable.

4. Comparison with SPN (Q1b)

Thank you for the suggestion. As SPN code is not available, we train SPN [R1] on MNIST and KMNIST and evaluate it under the same setup:

DatasetMethodFID ↓Precision ↑Recall ↑
MNISTFlow Matching0.8070.8360.937
VQ-VAE11.4680.9970.006
FSQ-VAE3.2110.7040.13
RQ-VAE2.820.970.367
SPN0.540.7380.95
Ours0.4220.810.949
KMNISTFlow Matching2.6360.7310.796
VQ-VAE2.3180.3770.94
FSQ-VAE6.2430.950.133
RQ-VAE2.3760.4490.941
SPN1.9040.6520.869
Ours1.6930.6430.81

5. Encoder training on ImageNet (Q2)

The encoder is trained through the SMM quantizer with STE. The STE gives approximately the same direction of gradient (Appendix Figure 8), therefore this choice does not substantially affect the results.

6. FSQ-VAE configuration (Q2)

FSQ-VAE has quantization levels L = (4, 4, 4, 4, 4, 4, 4), and it is trained with the same budget as SMM (50 epochs).

7. Comparison with LlamaGen and baselines in Tab. 7 (W2 & Q2)

Following LlamaGen [R6], we use the VQ-8 version. However, we reduce the number of latent channels from 8 to 7. We could not use the default 8-channel latent version due to the exponential explosion of the softmax number of choices. In Tab. 7, VQ-VAE uses LlamaGen setup (7-dim VQ-8 version) and is trained with the same protocol as SMM (50 epochs).

8. Sampling cost (Q2)

We report FLOPs per sample with KV caching:

MethodFLOPs per sample
Flow Matching21.3 TFLOPs (42.6 with CFG)
SMM0.55 TFLOPs (1.11 with CFG)

SMM requires about 39× fewer FLOPs per sample than Flow Matching, showing that SMM is more efficient at inference.

AuthorsOpenReview ↗

Response to Reviewer J2MM (2/2)

9. D3IL baselines (W2 & Q4)

Thank you for the suggestion. We retrain BESO and DDPM-ACT, and add VQ-BeT [R5]. Success rates (%), six-seed means:

MethodAvoidingAligningPushingSort-2Sort-4Sort-6Stack-1Stack-2Stack-3
BESO (retrained)29.882.376.981.231.00.189.069.19.0
DDPM-ACT (retrained)51.985.993.085.816.20.550.221.93.4
VQ-BeT73.373.177.774.713.30.412.30.30.0
SMM97.091.495.489.464.42.893.472.614.4

The retrained baselines are comparable to the published numbers on most tasks. SMM outperforms the retrained baselines and VQ-BeT on all nine tasks.

Regarding the lower behavior entropy on Pushing and Stack-1/2, our policy produces more predictable paths.

10. Marginal-CDF normalization (Q4)

We would like to clarify that the proof covers the general case. Prop. 1 is a worst-case bound over all x∈[0,1]Dx \in [0,1]^D, and Corollary 1.1 holds uniformly over all Borel probability measures on [0,1]D[0,1]^D. Neither makes assumptions on the data distribution or the normalization.

11. Thm. 3 and fractal targets (W2)

We agree that Thm. 3 does not apply to fractals. It applies to images with high dimensionality, where parallel refinement is used (Tab. 12 and Tab. 13) to compensate for high dimensionality.

[R1] Menick, J., & Kalchbrenner, N. (2019). Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference on Learning Representations.

[R2] Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., & Liu, X. (2025). Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15733-15744).

[R3] Morin, F., & Bengio, Y. (2005). Hierarchical probabilistic neural network language model. In International Workshop on Artificial Intelligence and Statistics (pp. 246-252). PMLR.

[R4] Li, T., Sun, Q., Fan, L., & He, K. (2025). Fractal generative models. Transactions on Machine Learning Research.

[R5] Lee, S., Wang, Y., Etukuru, H., Kim, H. J., Shafiullah, N. M. M., & Pinto, L. (2024). Behavior generation with latent actions. In International Conference on Machine Learning (pp. 26991-27008). PMLR.

[R6] Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., & Yuan, Z. (2024). Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.