acceptodds
Under review as a conference paper at ICLR 2027

Fourier Latent framework for 3D Point Cloud Generation

Abstract

Generating 3D point clouds poses unique challenges due to their unordered nature and variable cardinality. Existing generative approaches often couple point cloud shape representations to the number of points, limiting their flexibility and scalability. We propose FourierDiT, a Fourier latent framework that decouples shape representation from point sampling. Our key idea is to represent a point cloud through its underlying shape distribution using empirical characteristic-function evaluations at a set of learnable spatial frequencies. The resulting Fourier latent representation is permutation-invariant by construction and has a fixed dimensionality independent of point-cloud cardinality. Building on this representation, we train a shape-conditioned vector-field decoder via flow matching, allowing an arbitrary number of points to be sampled from a single latent code. A frequency-conditioned Diffusion Transformer further learns the distribution of Fourier latent codes, enabling the synthesis of novel shapes. Extensive experiments demonstrate competitive generation quality and improved training efficiency, while reconstruction and interpolation results highlight the effectiveness of the learned Fourier latent space. Our framework provides a flexible and efficient approach to 3D point cloud generation without tying the point cloud representation to a fixed number of points.

Primary area:
generative models
Keywords:
Point Cloud, Generative Model, Diffusion, Flow Matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_T4grOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to design a fixed dimensional representation for 3D point cloud sampled from a distribution and make the representation invariant to permutation and the number of points. The main approach is to create a Fourier representation by sampling random vectors and use Fourier positional encoding to encoded the projection of 3D points to these vectors. These features are decoded through a conditional flow matching in 3D space, which allows sampling of arbitrary number of points. The paper demonstrated that such features can be used in 3D point cloud generation, and showed strong performance comparable to some of the state-of-the-arts.

Critical strengths

The method is rather simple and easy to implemented. The paper is clearly written and generally easy to follow. The performance is decent and the ablation showed that under the current setup, the proposed fourier encoding is helpful.

Critical weaknesses
  • The model is not cardinality independence. While the dimensionality is independent of the number of the points, the latent depends the number of points entering into the encoder.
  • Validation results are mixed, and it does not demonstrate any unique advantage of the Fourier encoding.
  • It's not very clear to me what is the advantage of this proposed method comparing to training an VAE or AE that compressed the point cloud into a latent representation. While the authors have showed some baseline using similar technique, such as ShapeGF, such baseline is also trained with out-dated neural network architectures. It would be interesting to see what is the advantage of such Fourier feature encoding.
Questions for authors

See previous sections.

1 comment

We thank the reviewer for recognizing the simplicity and effectiveness of our approach, and for the insightful comments on the Fourier representation. We address these concerns below.

Cardinality independence of the representation.

Our formulation distinguishes the population-level Fourier representation from its empirical estimate. For a fixed frequency set KK and underlying distribution μ\mu, ΦK(μ)\Phi_K(\mu) is independent of the number of observed points. Under i.i.d. sampling from μ\mu, its empirical estimate zN=ΦK(μ^N)z_N=\Phi_K(\hat{\mu}_N) satisfies

zN→N→∞a.s.ΦK(μ).z_N \xrightarrow[N\to\infty]{\mathrm{a.s.}} \Phi_K(\mu).

Moreover,

\left\\|z_N-\Phi_K(\mu)\right\\|_F^2 \right] = \frac{1}{N}\sum_{m=1}^{M} \left(1-|\phi_\mu(k_m)|^2\right) \leq \frac{M}{N}.$$ For point clouds sampled from the same distribution, variation in the empirical coefficients reflects finite-sample estimation. Thus, increasing $N$ reduces the mean squared sampling error in the shape latents used for DiT training while preserving their dimensionality. Appendix D.1 empirically examines this sampling consistency, and Table 3 shows improved generation quality with denser training point clouds on Airplane. We will clarify this statistical interpretation in the revised manuscript. **Advantages of the Fourier representation.** A key advantage of the Fourier representation is that denser training point clouds reduce the sampling-induced estimation error in the shape latents used for DiT training, without increasing the latent dimensionality or the computational complexity of the DiT. This allows the latent generative model to learn from more accurate representations of the underlying shape distributions. Table 3 demonstrates that increasing the training cardinality improves generation quality with the same model architecture and latent dimensionality. Moreover, FourierDiT achieves competitive generation quality at substantially lower training costs than existing methods (Tables 1 and 2). The Fourier representation additionally provides an explicit spectral structure, with each latent token associated with a specific spatial frequency. Our frequency-conditioned DiT exploits this structure to model dependencies among Fourier coefficients. Table 4 demonstrates the benefits of learning the frequency locations and incorporating frequency information into the DiT. **Distinction from conventional AE/VAE representations.** FourierDiT explicitly parameterizes the shape latent through characteristic-function evaluations at learnable spatial frequencies. To evaluate its advantages over modern learned representations, we compare Fourier encoding with a PVCNN-based encoder on ShapeNet Airplane under the same latent dimensionality (256), training cardinality (15,000 points), and training time. For each validation shape, we repeatedly subsample the input point cloud and measure the relative latent error against its full 15,000-point representation, together with Top-1 retrieval accuracy among all reference shape latents. | Number of points | 128 | 256 | 512 | 1024 | 2048 | 4096 | 8192 | |---|---:|---:|---:|---:|---:|---:|---:| | PVCNN (Error ↓) | 1.933 | 1.275 | 0.706 | 0.356 | 0.190 | 0.114 | 0.065 | | Fourier (Error ↓) | 0.606 | 0.425 | 0.302 | 0.210 | 0.142 | 0.092 | 0.052 | | PVCNN (Acc. ↑) | 4.0% | 14.0% | 46.5% | 84.5% | 95.6% | 97.4% | 99.1% | | Fourier (Acc. ↑) | 65.6% | 81.3% | 90.8% | 96.2% | 98.3% | 99.5% | 100% | Fourier encoding consistently achieves lower representation errors and higher retrieval accuracy across all evaluated cardinalities, despite using substantially fewer encoder parameters (384 vs. 4.27M). These results demonstrate its ability to preserve shape-specific information under varying point sampling, providing more stable training representations for latent generative modeling.
Reviewer_76wQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FourierDiT, a two-stage framework for 3D point cloud generation. It represents point clouds using empirical characteristic-function evaluations at learnable frequencies, employs a flow-matching decoder for point sampling, and trains a frequency-conditioned Diffusion Transformer to generate shape latents. The method achieves competitive generation quality and reports reduced training costs.

Critical strengths

1.The two-stage framework is conceptually clean, and its generation results are competitive on standard ShapeNet benchmarks. 2.The paper includes analyses of learned frequencies, sampling consistency, and latent-space behavior.

Critical weaknesses

1.Limited novelty:The central idea of separating shape-level representations from arbitrary-cardinality point sampling has already been established by PointFlow (ICCV 2019) and FrePolad (ECCV 2024). The proposed Fourier encoder is essentially a finite-dimensional mean aggregation of sinusoidal features, while the flow-matching decoder and latent DiT follow established generative formulations. 2.The proposed formulation guarantees fixed latent dimensionality and permits arbitrary numbers of output particles. However, these properties do not imply cardinality-independent reconstruction fidelity or computational complexity.The appendix investigates input sampling consistency, but the paper lacks systematic quantitative evaluations of generation quality across output cardinalities using the same latent code. Furthermore, Fourier encoding still scales with the number of input points, and flow-based decoding scales with the number of generated particles. Consequently, the central claim of scalable, cardinality-independent generation is only partially substantiated. 3.The equations in the paper are not numbered and should be numbered.

Questions for authors

1.Limited novelty:The central idea of separating shape-level representations from arbitrary-cardinality point sampling has already been established by PointFlow (ICCV 2019) and FrePolad (ECCV 2024). The proposed Fourier encoder is essentially a finite-dimensional mean aggregation of sinusoidal features, while the flow-matching decoder and latent DiT follow established generative formulations. 2.The proposed formulation guarantees fixed latent dimensionality and permits arbitrary numbers of output particles. However, these properties do not imply cardinality-independent reconstruction fidelity or computational complexity.The appendix investigates input sampling consistency, but the paper lacks systematic quantitative evaluations of generation quality across output cardinalities using the same latent code. Furthermore, Fourier encoding still scales with the number of input points, and flow-based decoding scales with the number of generated particles. Consequently, the central claim of scalable, cardinality-independent generation is only partially substantiated. 3.The equations in the paper are not numbered and should be numbered.

1 comment

We thank the reviewer for appreciating our framework and latent-space analyses, and for the constructive feedback on novelty and scalability. We address these concerns below.

Novelty with respect to PointFlow and FrePolad.

FourierDiT builds on the established paradigm of shape-level generative modeling followed by point-level sampling, as exemplified by PointFlow and FrePolad, both cited in Sec. 2. Our contribution is the construction of the shape latent space through characteristic-function evaluations at learnable spatial frequencies, together with a latent generative model that exploits this explicit spectral structure.

The choice of sinusoidal features is motivated by the uniqueness theorem for characteristic functions, which establishes that the complete characteristic function uniquely determines a probability measure without requiring a density or finite moments. Our encoder constructs a finite-dimensional representation by evaluating this transform at spatial frequencies jointly optimized with the decoder.

This construction gives each latent token a precise spectral meaning and provides distribution-free estimation error bounds under i.i.d. sampling. These properties motivate our use of sinusoidal mean aggregation for shape representation. The ablations in Table 4 further support learning the frequency locations and incorporating the associated spatial frequencies into latent diffusion. We will make this representation-centered contribution more explicit in the revised manuscript.

Input cardinality and statistical consistency.

For a fixed frequency set KK, the population-level Fourier representation ΦK(μ)\Phi_K(\mu) depends only on the underlying distribution μ\mu. Under i.i.d. sampling from μ\mu, its empirical estimate zN=ΦK(μ^N)z_N=\Phi_K(\hat{\mu}_N) converges almost surely to ΦK(μ)\Phi_K(\mu) as NN increases, with a fixed dimensionality of M×2M\times 2.

This property allows denser training point clouds to reduce the sampling error in the shape latents used for DiT training while maintaining a fixed latent size. Appendix D.1 examines sampling consistency through repeated subsampling, and Table 3 shows that increasing the training cardinality from 2,048 to 15,000 improves generation quality on Airplane with the same model architecture and latent dimensionality. We will clarify this statistical interpretation in the revised manuscript.

Output cardinality.

Given a generated Fourier latent, the decoder defines a shape-conditioned distribution in continuous 3D space. Changing the output cardinality only changes the number of independent samples drawn from this distribution, leaving the learned conditional distribution unchanged. Flexible output cardinality follows from this established continuous-distribution-based generative paradigm that has been previously proposed. Our study focuses on improving shape representations for latent generative modeling through denser training point clouds. Accordingly, Table 3 investigates the effect of training cardinality while keeping the latent dimensionality fixed and adopting the conventional 2,048-point evaluation protocol for consistency with prior work on point cloud generation.

Computational scalability.

Our scalability argument concerns the cost of shape-level generative modeling. The DiT operates on a fixed sequence of MM Fourier tokens, independently of the input and output cardinalities. Moreover, Algorithm 1 separates the number of points used for Fourier representation estimation from the point batch size used for flow-matching training. Denser training observations can therefore be used with the same DiT sequence length and the same number of pointwise decoder evaluations per training iteration at a fixed point batch size.

Fourier encoding requires O(NM)O(NM) operations, and point decoding scales linearly with the output cardinality for a fixed latent and number of integration steps. The practical efficiency of our design is demonstrated in Table 2: the reported average training cost remains 48 GPU hours when the training cardinality increases from 2,048 to 8,192.

Equation numbering.

Thank you for the suggestion. We have numbered the relevant displayed equations in the revised manuscript.

Reviewer_KRhGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FourierDiT encodes a point cloud into a fixed-size latent by evaluating its empirical characteristic function at M=128 learnable 3D frequencies. The latent is permutation-invariant and independent of the number of points. A flow-matching decoder generates any number of points from the latent, and a DiT models the latent distribution. On ShapeNet, it reports competitive 1-NNA at 2,048 points and much lower training cost.

Critical strengths
  1. A simple, permutation-invariant, fixed-size representation, supported by appendix analyses (N^-1/2 error decay, frequency-dependent noise sensitivity).
  2. Competitive quality at much lower reported training cost, with ablations supporting the design choices.
Critical weaknesses
  1. The central claim, flexible cardinality, is not validated: all generation results use 2,048 points. There is also no comparison with a standard pooling encoder at the same latent size, so the benefit of the characteristic-function formulation is unclear.
  2. Gains are modest and mixed across categories and protocols, with no variance over seeds.
  3. Formatting: the text block appears wider than the official ICLR 2027 template, which would allow more content within the page limit. The authors should use the official template without modifying the margins or text width. I have also flagged this to the chairs.

I recommend weak reject (2) for now. I would raise it to 3 if the formatting is corrected and all the questions are answered.

Questions for authors
  1. Please confirm whether the paper uses the unmodified ICLR 2027 style file, and if not, please revise it to comply with the official format.
  2. How does generation quality change when sampling e.g. 1k, 8k and 15k points from the same latents?
  3. What happens if the encoder is replaced by a standard pooling encoder with the same latent size, decoder and DiT?
  4. How would the representation behave on real scans with non-uniform sampling density or partial observations, and do you expect a fixed M=128 to suffice for larger scenes? any plan or experiment about this? 5.A downstream evaluation such as training a classifier on generated samples (at different point counts) and testing on real data could.
2 comments

We thank the reviewer for recognizing the simplicity of our representation and its training efficiency, and for raising constructive questions. We address the concerns and questions below.

Scope of contribution.

Our contribution is a structured Fourier latent space within the established paradigm of shape-level generation followed by point-level sampling. A key advantage is that denser training point clouds reduce the sampling error in the shape latents used for DiT training without increasing the latent dimensionality or DiT sequence length. Appendix D.1 demonstrates the reduction in empirical representation error through repeated subsampling, while Table 3 shows improved generation quality with denser training point clouds at a fixed output cardinality.

Comparison with conventional pooling encoders.

Our choice of sinusoidal features is motivated by the uniqueness theorem for characteristic functions: the complete characteristic function uniquely determines a probability measure without requiring a density or finite moments. Evaluating this transform at learnable spatial frequencies provides a compact shape representation with explicit spectral structure, which is further exploited by our frequency-conditioned DiT (Table 4).

To evaluate its advantages over conventional pooling-based representations, we compare Fourier encoding with a DeepSets encoder using pointwise learned features followed by mean pooling on ShapeNet Airplane under the same latent dimensionality (256), training cardinality (15,000 points), and training time.

For each validation shape, we repeatedly subsample the input point cloud and measure the relative latent error against its full 15,000-point representation, together with Top-1 retrieval accuracy among all reference shape latents.

Number of points1282565121024204840968192
Mean pooling (Error ↓)0.6380.4470.3200.2200.1480.0990.056
Fourier (Error ↓)0.6060.4250.3020.2100.1420.0920.052
Mean pooling (Acc. ↑)32.4%50.8%68.2%85.2%94.7%98.0%99.2%
Fourier (Acc. ↑)65.6%81.3%90.8%96.2%98.3%99.5%100%

While both encoders exhibit the same empirical error decay rate, Fourier encoding consistently achieves lower representation errors and higher retrieval accuracy, with considerably fewer encoder parameters (384 vs. 99,328). These results demonstrate that Fourier encoding better preserves shape-specific information under finite-sample observations, providing more stable training representations for latent generative modeling.

Generation quality.

FourierDiT achieves competitive generation quality with a simple architecture and substantially lower training costs (Tables 1 and 2). On Airplane, Table 3 shows that increasing the training cardinality from 2,048 to 15,000 improves CD/EMD-based 1-NNA from 69.38/64.44 to 64.20/61.85 without introducing additional modules or increasing model complexity. This highlights the practical value of improving the statistical accuracy of shape representations for generative modeling.

Variance across random seeds.

Due to computational constraints, we report generation metrics without variance across random seeds, following common practice in the point cloud generation literature (e.g., PVD, LION, and TIGER). This convention and its computational motivation are explicitly discussed in the checklist of LION.

Formatting.

We have corrected the formatting to comply with the official ICLR 2027 style requirements, as reflected in the revised manuscript.

Input versus output cardinality.

Given a fixed shape latent, different output cardinalities correspond to different numbers of independent samples from the same learned conditional distribution. Our experimental focus is the effect of training cardinality on latent generative modeling. Accordingly, Table 3 varies the training cardinality while following the conventional 2,048-point evaluation protocol, enabling a controlled assessment of the benefit of denser training point clouds.

Non-uniform sampling and partial observations.

The current global Fourier encoder represents the observed sampling distribution. Non-uniform sampling therefore generally yields different Fourier coefficients from uniform sampling of the same surface, as densely sampled regions contribute more to the global average. For partial observations, the representation characterizes the observed portion of the shape.

A promising extension is to partition the 3D space into local regions, each represented by a dedicated set of spatial frequencies. Computing normalized Fourier coefficients independently within each region could mitigate the effects of sampling density variations across regions.

Choice of MM and scalability to larger scenes.

The choice of latent dimensionality is a general design consideration shared by latent generative frameworks, as more complex distributions may require greater representational capacity. In FourierDiT, the latent dimensionality is controlled by the number of frequencies MM, independently of the point-cloud cardinality NN. We adopt M=128M=128 for object-level generation based on the empirical trade-off between generation quality and training efficiency observed on Airplane in Table 5. The same formulation can be applied to more complex shape distributions with an appropriately chosen MM.

Downstream classification.

Downstream classification would complement our current evaluation by assessing the semantic utility of generated samples. The present study focuses on evaluating the Fourier latent space through distribution matching, representation consistency, and training efficiency. We consider downstream applications a natural direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.