acceptodds
Withdrawn from ICLR 2027

Grokking Geometry: The Phase Transition from Positional to Semantic Learning in 3D

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While 3D point cloud networks achieve high performance on complex benchmarks, it remains unclear whether they learn intrinsic geometric semantics or merely exploit spurious positional correlations. To address this, we introduce the Coordinate Trap, a synthetically controlled benchmark designed not to simulate ecological complexity, but to serve as a rigorous atomic unit test for geometric reasoning. By imposing a strong spatial bias (e.g., Sphere-Left, Cube-Right), we decouple absolute position from local topology. We observe that geometric generalization does not emerge gradually but follows a sharp phase transition governed by sample complexity (). We identify a critical threshold () separating two distinct regimes: a Positional Phase, where models minimize loss via coordinate memorization, and a Semantic Phase, where a symmetry breaking event occurs, realigning the decision boundary from embedding axes to intrinsic shape properties. Crucially, we distinguish this data-driven transition from optimization-time grokking: below , extended training fails to yield generalization, indicating that sufficient data density is a prerequisite for the optimization dynamics to escape the positional attractor. Furthermore, we quantify the excess sample complexity for generic architectures (e.g., Transformers) compared to those with geometric inductive biases, proposing as a novel metric for the geometric affinity of neural networks, providing a theoretical blueprint for scaling robust 3D foundation models.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
3D point clouds, geometric representation learning, shortcut learning, grokking, phase transitions

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_braJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the Coordinate Trap, a synthetic point-cloud benchmark with label-dependent spatial offsets during training and a reversed positional bias at test time. It studies how dataset size, architecture, and augmentation affect out-of-distribution generalization. Varying training size N from 32 to 16,384 across several architectures, the authors report that OOD accuracy jumps sharply at an architecture-specific critical size N_c. They interpret this as a symmetry-breaking transition from a positional to a geometric solution, distinct from optimization-time grokking. The paper further studies how architecture, optimizer, early stopping, and augmentation affect this threshold, proposes N_c as a measure of geometric affinity, and relates it to robustness against spatial offsets on ModelNet40.

Critical strengths
  • The controlled benchmark offers some sort of interpretable way to test whether a model relies on absolute position rather than shape. The overlapping training distributions and the reversed-bias evaluation make the shortcut explicit.
  • The paper includes optimizer comparisons, early-stopping experiments, a zero-bias control, additional shape pairs, and a ModelNet40 offset evaluation, which is useful.
Critical weaknesses
  • The central quantity N_c is reported inconsistently. Section 3.3 defines N_c as the sample size required to reach 50% OOD accuracy and reports N_c ≈ 4096 for PointNet, while Figure 1, 4, and 5 place the critical point at N = 1024. However, Table 2 reports an AdamW accuracy of 0.60 at N = 128 and an SGD accuracy of 0.99 at N = 32, and Table 3 also reports full-training accuracy above 50% at N = 128. By the paper’s own definition, N_c for PointNet would therefore be at most 128. Similarly, Section 3.2 states that sub-critical OOD accuracy remains “clamped at 0%,” whereas the lowest tabulated value is 0.26. These results must be reconciled before N_c can be interpreted as a stable, optimizer-independent property.

  • The geometric-invariance claims rest on qualitative evidence. The paper states that models above N_c acquire a “translation-invariant geometric function” (Figure 1) and that the learned attention pattern is “pose-invariant” (Figure 4), but invariance is never measured directly. The OOD test only reverses the position–label correlation, and showing saliency maps of the same object from three orthogonal viewpoints does not test invariance to pose.

Questions for authors
  • Can the authors reconcile the reported N_c values with Tables 2-3 and provide the underlying per-seed accuracy curves, fitting procedure, and bootstrap protocol? Are these results obtained under different augmentation or training settings?
  • Please provide per-seed OOD accuracy for all N and all architectures, the 5,000-epoch runs, and an explanation of the Table 2 vs. Table 3 discrepancy and the SGD result at N=32.
  • How do simple invariant shape-statistic baselines perform, and what additional geometric capability do the neural models demonstrate?
  • Does the transition remain sharp when dataset size and optimization-step budget are varied independently?
  • What evidence distinguishes the proposed basin-escape mechanism from ordinary improvements in generalization with more data?
Reviewer_khNKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interesting observation about the relationship between dataset size and the inductive biases of point cloud architectures. Using a synthetic binary classification task, the authors show that increasing the number of training samples can lead models to shift from relying on the point cloud centroid to learning translation-invariant representations. They identify a critical dataset size around which this transition occurs, exhibiting a grokking-like phenomenon in which the model shifts from exploiting centroid position to learning translation invariance. Experiments on a real-world dataset further reveal a correlation between this critical dataset size and robustness to translation perturbations: models that require more training samples to achieve translation invariance tend to exhibit lower robustness to translations.

Critical strengths
  • The use of a controlled experimental setting provides a useful framework for investigating the inductive biases of different point cloud architectures and their ability to generalize to OOD scenarios.
  • The observation that temporal grokking emerges only near a critical dataset size, while smaller datasets lead to shortcut learning is particularly interesting.
Critical weaknesses

Insufficient justification of the central claims and proposed mechanisms

  • The central argument that fposf_{\mathrm{pos}} is simpler than fgeof_{\mathrm{geo}} is not sufficiently justified. Citing Rahaman et al. (2019) does not establish this relationship. In what sense is fposf_{\mathrm{pos}} simpler? Since different architectures have different inductive biases, the notion of simplicity may itself be architecture-dependent. A more precise definition and justification are needed.

  • The discussion in Section 3.2 relies heavily on concepts such as optimization basins, saddle points, and residual gradient signals. However, despite mentioning them several times throughout the paper, these mechanisms are not investigated experimentally. In particular, the authors repeatedly mentioned gradient signals to explain the observed transitions without analyzing the gradients themselves. Hence, the proposed mechanistic explanation remains largely speculative.

  • Several interpretations of the experimental observations are overstated. For example, high training accuracy combined with poor OOD generalization does not necessarily imply exclusive memorization (L307), particularly when ID generalization remains good. Similarly, describing Figure 4 as a "visual proof" of "geometric grokking" is too strong. The visualization is limited to PointNet, and the observed saliency patterns do not establish that the learned representations are translation invariant.

  • The arguments in L418–431 are difficult to follow and not well established.

Inconsistencies in the experimental validation

  • The description in L304–345 does not appear fully consistent with Figure 5. For instance, the authors state that OOD accuracy remains at ~ 0% in the subcritical regime, whereas the figure appears to show an accuracy of approximately 25%.

  • The synthetic dataset could be described more precisely through a simple mathematical formulation of its generation process. Figure 2 is somewhat ambiguous: does each point represent the centroid of a point cloud? Moreover, although the text describes translations along the x-axis, the figure also shows variation along the y-axis.

  • The connection between the synthetic benchmark and robustness on real-world datasets is interesting but not sufficiently established. Table 1 demonstrates a correlation between NcN_c and sensitivity to translation perturbations. However, both quantities may largely reflect the degree of translation invariance already existing in the architectures. To demonstrate that NcN_c provides additional information beyond known architectural properties, it would be useful to investigate this relationship among architectures with similar invariance characteristics.

Some minor issues:

  • The organization of the methodology could be improved. In particular, Section 2.2 is difficult to follow without first understanding the dataset construction presented in Section 2.3. Reordering these sections would make the motivation and methodology easier to understand.

  • The paper appears to treat SE(3) invariance as a generally desirable property of point cloud representations. However, this assumption is task-dependent, as some 3D classification tasks may require positional or orientation information.

  • I acknowledge the interesting observation that temporal grokking may occur near a critical dataset size. However, I am not convinced that this transition should itself be characterized as "grokking". I would appreciate a stronger justification for this terminology.

Questions for authors
  • How do object scale and the number of points per object M affect the results? Could increasing M compensate for smaller N?
  • Could the authors provide the critical-point visualizations from Section 3.1 for other architectures?
  • Could the authors clarify the meaning of "symmetry breaking"? It is unclear which symmetry is broken, especially since the network appears to become more translation-invariant as N increases.
Reviewer_1qpUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Coordinate Trap, a synthetic point-cloud classification benchmark designed to expose positional shortcuts. The authors study how training-set size and architectural inductive biases affect out-of-distribution generalization, claim a critical sample threshold NcN_c associated with a transition from positional to geometric learning, and propose NcN_c as a measure of geometric robustness.

Critical strengths

The controlled benchmark provides an interpretable setting for examining positional shortcuts in point-cloud classifiers.

Critical weaknesses

W1. Fundamental inconsistencies in the reported results and the definition of NcN_c. Section 3.3 defines NcN_c using a 50% OOD-accuracy threshold and reports Nc≈4096N_c \approx 4096 for PointNet. However, Table 2 reports 60% OOD accuracy at N=128N=128 with AdamW and 99% at N=32N=32 with SGD, while Table 3 also reports accuracy exceeding 50% well below N=4096N=4096. The results in these tables are additionally inconsistent with the main-text curves. These discrepancies directly undermine the central empirical finding and the proposed metric.

W2. Insufficient justification of the benchmark's mathematical premise. Under the stated balanced class priors and uniform positional distributions, approximately 42% of the samples lie in the overlapping ambiguity region, where position alone cannot distinguish the classes. The optimal position-only classification accuracy is therefore approximately 79%, which does not support the claim that I(Y;Pos(X))≈H(Y)I(Y;\mathrm{Pos}(X)) \approx H(Y). The explanation that the ambiguity region is too sparsely populated at small NN to prevent an essentially perfect positional solution is consequently not established; near-zero training loss would require further explanation beyond a simple positional separator.

W3. The proposed phase-transition mechanism is largely speculative. A sharp-looking accuracy curve or a sigmoid fit does not establish a genuine phase transition, let alone symmetry breaking or an optimization trajectory escaping a positional attractor across a saddle point. The paper does not directly characterize the relevant loss landscape, gradients, or competing solutions, nor does it provide a derivation of a critical threshold. The observations could also reflect ordinary changes in generalization with sample size, so the mechanistic claims are substantially stronger than the evidence.

Questions for authors

Can the authors reconcile the NcN_c values and the conflicting results across Figures 5–6 and Tables 2–3, clarify the information-theoretic assumptions of Coordinate Trap, and provide direct evidence distinguishing the proposed basin-escape mechanism from ordinary sample-size-dependent generalization?

Reviewer_AsUZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether neural networks learn the actual geometry of 3D objects or rely on correlations between object positions and labels. The authors introduce a synthetic benchmark called Coordinate Trap, where spheres are mostly placed on one side and cubes on the other. During testing, they reverse the positions to see if models learned geometric information instead of positional shortcuts. They compare different point-cloud architectures and investigate how performance changes with the number of training samples. The authors claim that geometric learning emerges through a phase transition after reaching a critical sample size. They also study the effects of different architectures, data augmentation, and training duration, and evaluate the relationship between sample size and robustness on ModelNet40.

Critical strengths
  • The paper studies an interesting problem in geometric deep learning. The Coordinate Trap provides a simple way to investigate whether neural networks actually learn geometric information or just rely on positional shortcuts. Comparing different architectures and training-set sizes is also interesting and helps to understand how geometric inductive biases affect learning.
  • The authors include several experiments, such as different shape pairs, optimizer settings, training duration, and data augmentation. They also evaluate their findings on ModelNet40, which helps to connect the synthetic experiments to a more realistic setting.
Critical weaknesses
  • One of my main concerns is about the phase-transition claim, which is one of the main contributions of the paper. Although the results show a relatively sharp improvement in OOD accuracy as the number of training samples increases, I am not fully convinced that this represents a phase transition rather than a continuous improvement in generalization. The authors provide saliency visualizations and fit sigmoid curves, but I do not think these experiments are sufficient to establish the proposed symmetry-breaking mechanism. Also, it is unclear whether the reported critical sample size is a general property of the architecture or depends on the specific experimental setting.
  • The experiments are mostly based on a simple classification problem between spheres and cubes. Also, the authors frequently describe the learned features as topological invariants, but spheres and cubes have the same topology. Therefore, the main experiment seems to evaluate geometric rather than topological understanding. Another limitation is the ModelNet40 experiment, where the authors mainly investigate removing spatial centering. Since some architectures explicitly use relative coordinates, it is somewhat expected that they are more robust to translation. It would be interesting to see if sample size can predict robustness under other types of distribution shifts.
Questions for authors
  • How can the authors distinguish a phase transition from a sharp improvement in generalization as the training-set size increases? Can they provide additional evidence for the proposed symmetry-breaking mechanism?
  • Can the authors show whether sample size can predict robustness under other distribution shifts beyond spatial translation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.