acceptodds
Under review as a conference paper at ICLR 2027

PointSTAR: Continuous Next-Scale Autoregressive Generation for 3D Point Cloud

Abstract

Autoregressive point cloud generation is complicated by two properties of 3D geometry: point sets have no canonical ordering, and their coordinates are continuous. Existing methods address the first issue through manually ordered point sequences or coarse-to-fine next-scale prediction, but the latter commonly relies on vector-quantized tokens and therefore introduces a discrete bottleneck. We present PointSTAR, which formulates point cloud generation as continuous next-scale autoregressive prediction. PointSTAR first learns a compact hierarchy of progressively decodable continuous latents with a Nested-scale Variational AutoEncoder (NAVE). It then models generation at two complementary levels: inter-scale prediction captures coarse-to-fine structural evolution, while a multi-directional autoregressive flow captures dependencies among latent slots within each scale. This formulation preserves an explicit generation hierarchy without imposing a codebook on continuous geometry. Experiments on ShapeNet report competitive generation quality together with runtime comparisons, while reconstruction, interpolation, and component analyses examine the learned continuous hierarchy. PointSTAR also adapts to point cloud completion and upsampling, suggesting that the hierarchy is useful beyond unconditional generation.

Primary area:
generative models
Keywords:
3D point cloud generation, autoregressive generation, continuous-space generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8V6JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PointSTAR, a coarse-to-fine autoregressive framework for 3D point cloud generation that replaces discrete tokens with continuous latents to improve generation quality and efficiency.

Critical strengths
  1. Comprehensive experimental validation The paper provides a fairly complete set of experiments, including comparisons with recent SOTA methods (PointNSP and Quartet, 2026), extensive ablation studies, reconstruction, generation, and downstream tasks like point cloud completion and upsampling. Most results consistently outperform the baselines, which makes the proposed approach reasonably convincing.

  2. A simple but effective extension of next-scale autoregressive generation Replacing discrete tokens with continuous latents is a straightforward and well-motivated idea that avoids quantization errors. The paper also introduces a progressively decodable latent hierarchy and multi-directional autoregressive flows. Together, these design choices lead to consistent, though mostly modest, improvements in generation quality.

Critical weaknesses

Modest improvements over recent SOTA baselines.

Despite the extensive experiments, the improvements over PointNSP are generally small, often around 1–3 percentage points in generation metrics. The comparison with recent baselines is also incomplete, as several methods lack MMD and COV results. Given the additional complexity of the continuous latent modeling framework, the paper does not convincingly demonstrate a substantial enough improvement to justify its methodological contribution.

Questions for authors

You acknowledged that runtime comparisons are not fully standardized, but provide no similar justification for the missing generation metrics: do not explicitly explain why MMD and COV are missing for several recent baselines.

Is there any specific reason?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8V6J

Response to Reviewer 8V6J

We thank the reviewer for the positive evaluation of the experiments and the idea.

W1. Modest improvements over PointNSP. While the 1-NNA improvements over PointNSP on Airplane, Chair, and Car are relatively modest (1.5–2.7 points), we highlight three aspects that provide a more comprehensive assessment:

  • Improvements relative to the ideal 1-NNA. With 50% as the ideal reference, PointSTAR closes a substantial fraction of PointNSP's remaining gap, particularly on Chair and Car.
1-NNA-CD / EMD (%)AirplaneChairCar
PointNSP66.98 / 56.0554.01 / 53.7653.12 / 50.09
PointSTAR65.19 / 54.4452.45 / 51.0650.57 / 50.99
Gap to 50 closed11% / 27%39% / 72%82% / –

PointNSP retains a slight advantage on Car EMD.

  • MMD and COV show a clearer gap. With the newly completed metrics (Table R2 below), PointSTAR beats PointNSP on every MMD and COV entry. For example, MMD-CD on Chair is 2.329 vs. 3.774, and COV-EMD on Chair is 53.17 vs. 46.18.
  • The margin grows on harder settings. On ScanNet scenes the gap to PointNSP is 15.3 / 24.2 points (76.04/66.51 vs. 91.35/90.74). On the fine-structure categories Bookshelf, Faucet and Motorcycle, the mean margin over PointNSP is 4.63 / 2.63, compared with 1.97 / 1.14 on Airplane, Chair and Car. At 8192 points this margin grows to 9.23 / 7.53 (G2).

The extra components are also not costly in practice. PointSTAR samples in 1.37 s end-to-end, compared with 3.59 s for PointNSP and 5.45 s for Quartet as reported, and trains in 46 GPU hours including NAVE.

W2 / Q1. Why are MMD and COV missing for several baselines? Those methods (e.g., PointNSP, Quartet, FrePoLad, PSF) report only 1-NNA in their papers, and we copied published numbers without estimating the missing metrics ourselves. 1-NNA was preferred because, as discussed in PointFlow (Yang et al., 2019), MMD is insensitive to low-quality samples and COV to sample quality. 1-NNA reflects both fidelity and diversity. Following your suggestion, we evaluated TIGER, FrePoLad and PointNSP ourselves with the same reference sets, normalization and evaluation code as Table 3. No pretrained checkpoints are released for these methods, so we retrained each of them on the same training split with its public code and the code's default hyperparameters, without additional tuning. For FrePoLad, the defaults in the released code differ from the settings described in the paper, so its MMD/COV may be somewhat pessimistic. We have noted this in the revision (Appendix E.1). We then evaluated the samples with the same reference sets, normalization and evaluation code as Table 3. For consistency with Table 3, the 1-NNA column in Table R2 reports the values published in the original papers, MMD and COV come from our retrained models.

Table R2. Completed metrics. MMD-CD ×10³, MMD-EMD ×10², COV and 1-NNA in %. Each cell is CD / EMD.

ClassMethodMMD ↓COV ↑1-NNA (→50)†
AirplaneTIGER0.225 / 0.36146.17 / 51.1167.21 / 56.26
FrePoLad0.319 / 0.59746.91 / 44.2665.25 / 62.10
PointNSP0.285 / 0.42246.45 / 47.7666.98 / 56.05
PointSTAR0.209 / 0.33850.86 / 53.5765.19 / 54.44
ChairTIGER2.577 / 1.54546.37 / 51.8154.32 / 51.71
FrePoLad3.734 / 3.46342.57 / 41.6352.35 / 53.23
PointNSP3.774 / 1.85247.19 / 46.1854.01 / 53.76
PointSTAR2.329 / 1.47850.45 / 53.1752.45 / 51.06
CarTIGER1.142 / 0.78346.88 / 50.8554.12 / 50.24
FrePoLad2.931 / 1.33445.56 / 46.2751.89 / 50.26
PointNSP1.451 / 1.21746.42 / 48.6453.12 / 50.09
PointSTAR0.873 / 0.74451.14 / 53.1350.57 / 50.99

† For baselines, 1-NNA values are taken from the original papers, as in Table 3. MMD and COV for TIGER, FrePoLad and PointNSP are computed from our retrained models.

We can see that PointSTAR obtains the best MMD and COV on all 12 entries against these three baselines. Against PointNSP specifically, MMD-CD is 27–40% lower and COV is 3.3–7.0 points higher on every category. Taking the full Table 3 into account, PointSTAR has the best MMD and COV in 11 of 12 entries. The exception is Car COV-EMD, where LION reaches 56.53 vs. our 53.13. Note also that on Car, where PointNSP has the better 1-NNA-EMD (50.09 vs. 50.99), PointSTAR is still clearly better on MMD-EMD (0.744 vs. 1.217) and COV-EMD (53.13 vs. 48.64). Quartet, PSF and not-so-ot-flow also release no checkpoints. Retraining them was not feasible within the rebuttal period, and we will add their MMD/COV in the revision. We have added these rows to Table 3 of the revised paper, marked with † as evaluated by us.

Reviewer_iAbHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PointSTAR, a continuous next-scale autoregressive framework for 3D point cloud generation. It first learns a hierarchical continuous representation using a Nested-scale Variational Autoencoder, with progressive flow-matching decoders that reconstruct complete point clouds at increasing resolutions. With NAVE frozen, the generative model learns the latent distribution through conditioning on preceding scales and autoregressive normalizing flows within each scale. Different causal orderings across flow blocks model intra-scale dependencies, while multi-noise training and score-based refinement support sampling. Experiments on ShapeNet evaluate reconstruction and generation, examine individual components through ablations, and demonstrate applications to point cloud completion and upsampling.

Critical strengths

(1) Well-motivated continuous hierarchical representation. NAVE explicitly supervises each latent prefix to reconstruct a complete shape at increasing resolutions. This provides meaningful coarse-to-fine generation targets while avoiding a finite vector-quantization codebook.

(2) Coherent modeling of structural dependencies. The framework combines cross-scale conditioning with intra-scale autoregressive modeling. Varying causal orderings across flow blocks allows latent slots to receive different contexts, while the invertible transformations enable tractable likelihood-based training.

(3) Competitive empirical results. NAVE improves over PointNSP’s VQ-VAE on both reconstruction metrics across the three evaluated categories. The full framework also achieves competitive generation results, particularly on Chair and Car.

(4) Useful ablations and downstream demonstrations. The paper examines scale count, cross-scale conditioning, causal ordering, decoder design, and refinement budget. Completion and upsampling experiments further demonstrate the framework’s applicability beyond unconditional generation.

Critical weaknesses

W1. Limited Evaluation of Complex Geometries and Generalization.

Although the model is trained across 55 categories, the quantitative multi-category evaluation in Table 4 focuses on only three categories. It remains unclear how well the method handles more challenging geometries, particularly objects with thin structures or complex local details. A broader evaluation would better demonstrate the generalization capability of the proposed approach.

W2. Limited Methodological Novelty.

While the overall framework is interesting, several core components are closely related to existing methods, such as next-scale prediction in PointNSP and autoregressive flows in TARFlow/iTARFlow. The main contribution appears to lie in integrating these existing techniques into a continuous point cloud generation framework. The methodological novelty beyond these prior works could be more clearly demonstrated.

Questions for authors

Q1. Could the authors provide additional quantitative evaluations across more categories, particularly those involving complex geometries or fine-grained structures, to better demonstrate the generalization capability of PointSTAR?

Q2. Could the authors further clarify the key methodological differences between PointSTAR and existing approaches such as PointNSP and TARFlow/iTARFlow? In particular, what are the main technical innovations beyond combining continuous latent representations with existing autoregressive flow models?

1 comment
AuthorsOpenReview ↗

Response to Reviewer iAbH

Response to Reviewer iAbH

We thank the reviewer for the positive assessment of NAVE and of the cross-/intra-scale modeling, and for the concrete questions.

W1 / Q1. Evaluation on complex geometries and more categories. We agree that three categories are not enough to assess generalization to thin or detailed structures. We added three experiments (details and tables in G2):

  1. All 55 categories grouped by geometric complexity. Grouping uses the median Sphere Ratio (surface area relative to a volume-matched sphere, higher means thinner, less compact). Low (18) / Medium (18) / High (19) groups are evaluated with the 55-category model and compared against PVD and PointNSP (Table G2). PointSTAR is best in all groups, and its margin over PointNSP grows with complexity: 1-NNA-CD / EMD gains of 1.06 / 0.23 for Low, 2.12 / 1.53 for Medium and 4.09 / 2.81 for High. PointSTAR is also the most robust: from Low to High it degrades by 4.79 / 1.21 points, compared with 7.82 / 3.79 for PointNSP.
  2. Three fine-structure categories (Table G3): Bookshelf (thin plates and partitions), Faucet (thin curved tubes) and Motorcycle (many thin, connected parts). PointSTAR is best on all six entries. Its margin over PointNSP (CD / EMD) is 4.79 / 3.43, 2.60 / 1.19 and 6.50 / 3.26, a mean of 4.63 / 2.63. On the original Airplane / Chair / Car, the mean margin is 1.97 / 1.14. Motorcycle, the category with the most thin, connected parts, shows the largest gain. At 8192 points the mean margin grows to 9.23 / 7.53 (Table G3b), and PointSTAR degrades far less than PointNSP when the resolution increases (1.37 / 1.29 vs. 5.97 / 6.19).
  3. Higher resolution and real scenes (Tables G4–G5): 8192-point ShapeNet generation, where PointSTAR obtains the best 1-NNA on all reported entries, and ScanNet scene generation (76.04/66.51 vs. 91.35/90.74 for PointNSP).

On the same fine-structure categories, our complementary ordering also gives larger gains over random ordering than on the original three (mean 2.82/2.81 vs. 1.29/0.66, see Table R1 in our response to juBV). This suggests the intra-scale design specifically helps with multi-part, thin geometry.

W2 / Q2. Methodological differences from PointNSP and TARFlow/iTARFlow. Please see G1 for a side-by-side comparison. In short:

  • vs. PointNSP: we replace residual-VQ tokens with a prefix-decodable continuous hierarchy (NAVE), and categorical next-scale prediction with an exact-likelihood continuous model that also captures within-scale dependencies. PointNSP predicts the tokens of a scale in parallel.
  • vs. TARFlow/iTARFlow: these model a single flat image sequence. PointSTAR nests AR flows inside a next-scale factorization: coarser scales parameterize the base distribution and condition every block. It also uses complementary multi-directional orderings, rather than two alternating directions, to rotate context-poor positions across slots.
  • Relationship to prior work: PointSTAR adopts causal Transformer-based flow blocks and iTARFlow-style multi-noise training and score refinement, while introducing the above representation and modeling designs for hierarchical 3D point cloud generation.

These distinctions are validated by ablations (Table 2(B): continuous vs. capacity-matched or larger quantized, Table 6(A): cross-scale guidance, Table 6(B) and Table R1: ordering).

Reviewer_juBVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PointSTAR, a continuous next-scale autoregressive framework for 3D point cloud generation. It introduces a Nested-scale Variational AutoEncoder (NAVE) to learn progressively decodable continuous latent representations and employs inter-scale conditioning and multi-directional intra-scale autoregressive flows to model their joint distribution. Experiments on ShapeNet demonstrate competitive generation and reconstruction performance, with additional evaluations on point cloud completion and upsampling.

Critical strengths
  • The paper addresses a meaningful limitation of existing next-scale autoregressive point cloud generation methods by replacing discrete vector-quantized representations with continuous latent variables.
  • The overall framework is well motivated, and the combination of progressive representation learning and hierarchical autoregressive modeling is technically coherent.
  • The paper presents relatively comprehensive experiments, including comparisons with representative generative models, reconstruction and generation evaluations, component ablations, and extensions to completion and upsampling.
Critical weaknesses
  • The next-scale prediction framework is closely related to PointNSP, while multi-directional autoregressive flows and multi-noise refinement build upon TARFlow and iTARFlow, respectively. Although their integration into continuous point cloud generation is meaningful, the paper does not sufficiently establish the fundamental technical differences from these existing approaches or demonstrate why the proposed combination is necessary.
  • While the ablations support continuous representations and hierarchical modeling, the comparisons between continuous and quantized latents do not fully control representation capacity and model complexity. Moreover, multi-directional ordering provides relatively modest improvements over random multi-directional ordering (Table 5), leaving the necessity of the specific ordering strategy insufficiently justified.
  • The reported training and inference costs are not established under fully unified experimental settings, making the claimed efficiency advantages difficult to assess fairly. Furthermore, the experiments primarily focus on fixed-resolution object-level point clouds from ShapeNet. The effectiveness of the framework on more complex geometries and large-scale 3D scenes remains unclear.
Questions for authors

Please see the Weaknesses for details.

1 comment
AuthorsOpenReview ↗

Response to Reviewer juBV

Response to Reviewer juBV

We thank the reviewer for the detailed comments.

W1. Technical differences from PointNSP / TARFlow / iTARFlow, and necessity of the combination. Please see G1 for a detailed comparison. PointSTAR introduces three distinct technical contributions: (1) the prefix-decodable NAVE hierarchy, (2) hierarchical continuous likelihood modeling with cross-scale conditioning of both the Gaussian base and flow blocks, and (3) complementary multi-directional autoregressive ordering. Their effectiveness is independently supported by ablations:

  • continuous vs. quantized latents at matched or larger capacity: Table 2(B);
  • cross-scale guidance (None → Z1Z_1 → Z1:2Z_{1:2}: 63.18/62.44 → 59.43/56.92 → 56.07/52.16): Table 6(A);
  • ordering strategy: Table 6(B) and W2 below.

We have added the discussion regarding G1 comparison table to the revised related work and Appendix B.

W2a. Capacity and model complexity in the continuous vs. quantized comparison.

  • Model size. Stage-1 networks share the same depth. The continuous NAVE has 45.40M parameters, and the quantized variants have 45.73M–47.34M because of the extra codebook. The continuous model therefore has no size advantage.
  • Capacity. Continuous and quantized latents have no exact capacity equivalence, so Table 2(B) keeps the continuous model fixed (N = 21 slots) and enlarges the quantized one along both axes: codebook size K and number of latent units N. Even when the quantized variant reconstructs better (CD/EMD 3.38/4.13 vs. 3.68/4.34), it generates clearly worse samples (1-NNA-CD/EMD 65.91/62.83 vs. 56.07/52.16).

This indicates that the advantage is not explained by limited quantized capacity. We have described this design rationale explicitly in the revision.

W2b. Necessity of the specific ordering. We agree that the averaged gain over random multi-direction ordering in Table 6 (1.29 / 0.66) is modest. Per-category results show where it comes from.

Table R1. 1-NNA-CD / EMD (%), random multi-direction vs. ours.

AirplaneChairCarMean
Random multi-dir.65.54 / 54.6454.60 / 52.6151.94 / 51.22
Ours65.19 / 54.4452.45 / 51.0650.57 / 50.99
Gain0.35 / 0.202.15 / 1.551.37 / 0.231.29 / 0.66
BookshelfFaucetMotorcycleMean
Random multi-dir.52.98 / 57.5562.23 / 59.3754.77 / 52.68
Ours50.16 / 53.2760.33 / 57.3351.03 / 50.58
Gain2.82 / 4.281.90 / 2.043.74 / 2.102.82 / 2.81

The improvement is modest on Airplane but more pronounced on Chair, and roughly 2–4× larger on thin, multi-part categories (mean: 2.82/2.81 vs. 1.29/0.66). This trend aligns with our motivation: while random orderings diversify the autoregressive context, they do not balance which latent slots occupy context-limited early positions across blocks. In contrast, our complementary orderings explicitly distribute these positions across latent slots, yielding greater benefits for geometrically complex shapes requiring coordination across multiple parts. We have included these per-category results in the revised paper (Sec. 4.6 and Appendix E.3, Table 11).

W3a. Efficiency under unified settings. We clarify the timing scope. The reported 1.37 s/sample is end-to-end: it includes per-scale latent sampling, the ten latent-refinement steps per scale, and NAVE decoding to points. The 46 GPU hours cover both NAVE and generator training. We acknowledge that the baseline costs in Table 3 are taken from the respective papers and are not measured under one setting. In the revision (Sec. 4.1, Sec. 4.4 and Appendix A.3), we have (i) described the timing scope and hardware explicitly, and (ii) presented the efficiency comparison as indicative, not as a strict ranking. The efficiency argument does not rest on exact numbers. With ten refinement steps per scale, PointSTAR generates a shape from a small number of continuous latent slots per scale (21 slots in total). Point-space diffusion models such as PVD, by contrast, denoise all 2,048 points at every one of many steps.

W3b. Complex geometries, resolution and scenes. We added (G2) a 55-category complexity analysis, three fine-structure categories, 8192-point ShapeNet generation, and unconditional ScanNet scene generation. On ScanNet, PointSTAR reaches 1-NNA-CD/EMD 76.04/66.51, compared with 83.01/81.41 for PVD and 91.35/90.74 for PointNSP.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised Paper

We thank all reviewers again. All changes in the revised PDF are highlighted in blue.

  • Relation to prior work (juBV, iAbH): new paragraph in Related Work; side-by-side comparison with PointNSP and TARFlow/iTARFlow in Appendix B (Table 9).
  • Continuous vs. quantized capacity and model size (juBV): design rationale and parameter counts added to Sec. 4.3.
  • Complete MMD/COV (8V6J): TIGER, FrePoLad and PointNSP added to Table 3; retraining protocol in Appendix E.1.
  • Efficiency (juBV, 8V6J): end-to-end timing scope and hardware stated in Sec. 4.1, Sec. 4.4 and Appendix A.3.
  • Complex geometry and scenes (iAbH, juBV, 8V6J): new Sec. 4.5 with Table 5. It covers (A) Sphere-Ratio complexity groups over all 55 categories, (B) Bookshelf, Faucet and Motorcycle, and (C) ScanNet scenes. Details are in Appendix E.2 and E.5, with samples in Figures 9 and 11.
  • 8192-point generation (juBV, iAbH): Appendix E.4 (Table 12, Figure 10), including the fine-structure categories.
  • Per-category ordering ablation (juBV): Sec. 4.6 and Appendix E.3 (Table 11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.