- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces the Coordinate Trap, a synthetic point-cloud benchmark with label-dependent spatial offsets during training and a reversed positional bias at test time. It studies how dataset size, architecture, and augmentation affect out-of-distribution generalization. Varying training size N from 32 to 16,384 across several architectures, the authors report that OOD accuracy jumps sharply at an architecture-specific critical size N_c. They interpret this as a symmetry-breaking transition from a positional to a geometric solution, distinct from optimization-time grokking. The paper further studies how architecture, optimizer, early stopping, and augmentation affect this threshold, proposes N_c as a measure of geometric affinity, and relates it to robustness against spatial offsets on ModelNet40.
Critical strengths
- The controlled benchmark offers some sort of interpretable way to test whether a model relies on absolute position rather than shape. The overlapping training distributions and the reversed-bias evaluation make the shortcut explicit.
- The paper includes optimizer comparisons, early-stopping experiments, a zero-bias control, additional shape pairs, and a ModelNet40 offset evaluation, which is useful.
Critical weaknesses
-
The central quantity N_c is reported inconsistently. Section 3.3 defines N_c as the sample size required to reach 50% OOD accuracy and reports N_c ≈ 4096 for PointNet, while Figure 1, 4, and 5 place the critical point at N = 1024. However, Table 2 reports an AdamW accuracy of 0.60 at N = 128 and an SGD accuracy of 0.99 at N = 32, and Table 3 also reports full-training accuracy above 50% at N = 128. By the paper’s own definition, N_c for PointNet would therefore be at most 128. Similarly, Section 3.2 states that sub-critical OOD accuracy remains “clamped at 0%,” whereas the lowest tabulated value is 0.26. These results must be reconciled before N_c can be interpreted as a stable, optimizer-independent property.
-
The geometric-invariance claims rest on qualitative evidence. The paper states that models above N_c acquire a “translation-invariant geometric function” (Figure 1) and that the learned attention pattern is “pose-invariant” (Figure 4), but invariance is never measured directly. The OOD test only reverses the position–label correlation, and showing saliency maps of the same object from three orthogonal viewpoints does not test invariance to pose.
Questions for authors
- Can the authors reconcile the reported N_c values with Tables 2-3 and provide the underlying per-seed accuracy curves, fitting procedure, and bootstrap protocol? Are these results obtained under different augmentation or training settings?
- Please provide per-seed OOD accuracy for all N and all architectures, the 5,000-epoch runs, and an explanation of the Table 2 vs. Table 3 discrepancy and the SGD result at N=32.
- How do simple invariant shape-statistic baselines perform, and what additional geometric capability do the neural models demonstrate?
- Does the transition remain sharp when dataset size and optimization-step budget are varied independently?
- What evidence distinguishes the proposed basin-escape mechanism from ordinary improvements in generalization with more data?