acceptodds
Under review as a conference paper at ICLR 2027

Beyond Permutation Invariance: Idempotence in Set Data Representation

11 views

Abstract

Permutation-invariant architectures are routinely described as learning on sets, but permutation invariance removes order without removing multiplicity. This distinction is often hidden because datasets typically provide only one finite registration of each underlying datum. If repeated registration of an already-present element does not change that datum, its representation must also be consistent across such alternative registrations. We formalize this requirement as idempotent set consistency and prove that a variable-cardinality family descends to a function on finite subsets exactly when it is permutation invariant and compatible with duplicate insertion and deletion. This yields registration fibers and quantitative fiber defects that distinguish set semantics from multiset and empirical-measure semantics. More generally, we use directed systems of registrations as a semantic checklist: the chosen cross-cardinality maps specify which changes of registration are intended to preserve the datum, and their compatibility reveals which invariances a representation should satisfy. For set semantics, this perspective leads naturally to Hausdorff geometry and near-idempotence under arbitrarily close observations. We study these ideas through pooling and Set Transformer audits, geometry-controlled interventions on pretrained ModelNet40 classifiers, and duplicate and near-duplicate corruptions in human trajectory prediction. Rather than treating idempotence as a universal desideratum, our framework uses registration consistency to distinguish competing data semantics and tests whether a model respects the one intended by the application.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
permutation invariance, idempotence, set representation, set geometry, representation semantics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pvkZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the mismatch between permutation symmetries and the symmetries of sets, specifically the multiplicity of elements in contrast to multisets. The authors argue that permutation invariance is insufficient when true sets are desired (unordered and multiplicity invariant) and propose definitions to characterize this notion as idempotent set consistency. The extent to which several existing methods are sensitive to (near) duplicate insertion is measured in three experiments.

Critical strengths

Interesting problem

I think this is an interesting problem to study. Similar to how there are subtleties between the interaction of permutation symmetry and multisets (multiple copies allowed, the authors make a case that the same is true for permutation symmetry and sets (multiple copies disallowed) and I would be inclined to agree. The study of how exactly the multiplicity affects how sets should be modeled is interesting purely from a theoretical level alone, and the paper makes a good start in characterizing the problem, while also touching on the important aspect that continuity should be respected.

This point is consequential because I believe that the the topic itself would be of interest to ICLR audiences.

Critical weaknesses

Insufficient depth

I was disappointed to see a lack of insight into how the formalization can be used beyond a simple evaluation of the proposed metrics on trained models. As it stands, it appears to be an empirical metric that cannot be used a-priori to judge whether a model architecture is suited for satisfying the properties the authors define. It is presented as a purely post-hoc analysis of how robust some models are to different multiplicities, making it significantly less applicable to practical scenarios.

There does not seem to be any attempt made at discussing whether the stability to duplicate elements can be guaranteed through architecture. Such a discussion is usually a core aspect of papers in the field. Given the inherently discontinuous objective here, achieving it seems tricky, so this is a major omission in my view.

The main conclusion seems to be essentially "you should measure to what extent models are sensitive to duplicates if you care about duplicates", which is a fairly obvious statement. I did not get any larger insights than that from my reading of the paper.

This point is consequential in that in my opinion, the lack of depth of the study makes it insufficient to clear the bar of an ICLR submission. It feels more appropriate as a submission to a workshop, and after the ideas in the paper (which are a good start) have been fleshed out further, it might qualify for a conference in the future.

Questions for authors

My main question is in terms of any real-world application of the proposed method:

  • To what extent are duplicate observations realistic? With the example in the paper: duplicate pedestrian registration is only duplicate if all identifying information is exactly equal too (i.e. position, size, but also contents of registered pedestrian are equal). It seems easy to just pre-process such cases away if they are undesirable. Why should we rely on the model for that? With higher dimensionality of set elements it also seems less likely for elements to end up exactly equal.
6 comments
AuthorsOpenReview ↗

Response to Reviewer pvkZ (1/2): Architectural guarantees and contribution

Thank you for recognizing the relevance of multiplicity and continuity.

it appears to be an empirical metric that cannot be used a-priori to judge whether a model architecture is suited for satisfying the properties the authors define

Proposition D.1 in Appendix D.2 gives an architectural sufficient condition: shared pointwise features, coordinatewise max pooling, and a decoder, F(A)=ρ(max⁡a∈Ah(a)).F(A)=\rho \left(\max_{a\in A}h(a)\right). This construction is exactly duplicate-consistent. If the feature coordinates are LhL_h-Lipschitz and the decoder is LρL_\rho-Lipschitz from the maximum norm, then dY(F(A),F(B))≤LρLhdH(A,B).d_Y(F(A),F(B))\le L_\rho L_h d_H(A,B). These conditions can be assessed before training. They guarantee both exact consistency and a cardinality-independent bound for nearby supports.

It is presented as a purely post-hoc analysis of how robust some models are to different multiplicities

The experiments evaluate trained models, but Proposition D.1 applies to an architectural class independently of the empirical audit. The paper therefore provides both an a priori sufficient condition and a post-training assessment of existing implementations.

There does not seem to be any attempt made at discussing whether the stability to duplicate elements can be guaranteed through architecture.

This discussion is already present in §6 and Appendix D. D.2 proves the max-based Hausdorff bound above; D.6 connects it to the geometric audit; D.7 gives a hard-neighborhood discontinuity example; D.8 establishes conditional stability for an integral representation; and D.9 distinguishes structural guarantees from learned approximate consistency. We will bring the central result forward in the main text and ask that these existing sections be considered when assessing this point.

The additional Social-STGCNN analysis provides a concrete architectural diagnosis. Its off-diagonal graph rule assigns zero to identical observed displacement vectors and inverse distance to distinct vectors. With two nodes, self-loops of weight 1, and r=∥v1−v2∥r=\|v_1-v_2\|, the normalized Laplacian is

\begin{pmatrix} 1 & -1 \\\\ -1 & 1 \end{pmatrix} \quad(r>0),\qquad L(0)=0.$$ Thus this graph representation is discontinuous at coincidence, independently of trained weights. The accompanying noise sweep measures forecast sensitivity at finite scales; it does not by itself prove a limiting jump in the trained predictor. > Given the inherently discontinuous objective here, achieving it seems tricky Could you clarify which objective you regard as "inherently discontinuous"? > The main conclusion seems to be essentially "you should measure to what extent models are sensitive to duplicates if you care about duplicates" Our conclusion concerns **geometric set data representation**: the datum is an underlying geometric object, and its observed points are finite approximations of that object. For a CAD shape, the surface exists independently of the chosen sample size or sampling density. A representation intended for that shape should therefore respect its geometry across different acquisitions. Exact idempotence is the zero-discrepancy case; near-idempotence addresses close, nonidentical observations. This exposes a bias in the benchmark protocol. Fixed-cardinality, uniformly sampled training and evaluation can reward models specialized to that acquisition law while leaving geometric consistency untested. High benchmark accuracy can therefore coexist with substantial representation changes when the same object is sampled differently. Our audits investigate this gap between learning the sampled registration and representing the geometric object. We will revise the title to "Beyond Permutation Invariance: Idempotence in Geometric Set Data Representation" and shift the main emphasis from exact idempotence to near-idempotence: stability under geometrically close registrations of the same object.
AuthorsOpenReview ↗

Response to Reviewer pvkZ (2/2): Practical relevance and near-duplicate forecasts

To what extent are duplicate observations realistic?

Duplicate detections and tracks remain a documented problem in modern tracking. Beyond UMOT's comparisons with MOTR and MOTRv2 already cited in §5.3, two relevant examples are:

  • MCTR, WACV 2025 Workshops, §4 reports that MOTR, evaluated independently in each camera view, produces highly overlapping duplicate detections and consequently duplicate tracks. Its authors add NMS with an IoU threshold of 0.95 to address this observed failure.
  • TrackTrack, CVPR 2025 introduces Track-Aware Initialization specifically to suppress new tracks from detections that overlap active tracks or more confident detections.

The first documents an actual duplication failure; the second shows that preventing redundant track initialization remains an active design problem in a recent tracker.

duplicate pedestrian registration is only duplicate if all identifying information is exactly equal too

The relevant redundancy is that multiple records refer to the same physical pedestrian. Their estimated positions or histories need not be numerically identical. Our audited inputs are eight-step position histories, and the inserted records have a known source pedestrian. Exact copying provides the reference case; perturbing those copies tests redundant but slightly inconsistent observations.

It seems easy to just pre-process such cases away if they are undesirable. Why should we rely on the model for that?

UMOT's comparisons and MCTR's reported duplicate tracks show that redundant registration persists in evaluated tracking systems; TrackTrack explicitly adds a mechanism to suppress redundant track initialization. These examples show that preprocessing is already part of the solution, and that association of nearby, nonidentical tracks remains a substantive problem. Removing identical arrays alone does not resolve it. Our near-idempotence audit asks how a forecasting model responds when residual redundant records reach it. This makes downstream geometric stability a practical requirement alongside upstream duplicate suppression.

Tables 25–27 already include σ=0.02\sigma=0.02 and 0.050.05 m. With four added records, the new mean forecast displacements are:

σ\sigma (m)Social GANSocial-STGCNNAgentFormer
0.020.01300.21800.1291
0.050.03010.21090.1331

These average all 20 forecasts and 12 prediction steps for original pedestrians, with equal averaging across the five ETH/UCY folds. Social GAN and Social-STGCNN use paired latent and standard-normal draws; AgentFormer recomputes its native deterministic proposals, paired by index.

The results establish sensitivity to near-duplicate histories, including with native proposal generation. The controlled noise scales are not estimates of real tracker-error frequencies; the tracking references establish why the redundant-record failure mode itself is practically relevant.

Thank you for the clarifications and pointing me to the relevant statements in Appendix D, which I had missed.

The reframing towards near-idempotence is exactly addressing the discontinuity question I had. I believe that this change in perspective and the expanded motivation in the response on geometric set data representation makes the motivation for studying this problem clearer and would be a positive change for the positioning of the paper.

In my view, the contribution is still somewhat incremental and the contributions don't fit together neatly yet:

The paper therefore provides both an a priori sufficient condition and a post-training assessment of existing implementations.

The key aspect that is missing for me is the connection from the a priori sufficient condition (which I find fairly weak by itself) and the experiments, mostly on models that have relatively little connection to it. I would have liked to see more impactful insights than the condition about max pooling, which is a known result as other reviewers have pointed out, and how that would inform model design.

In its current state, it does not meet the bar for acceptance for me, but I encourage the authors to study this interesting problem further to gain deeper insights first, which would significantly strengthen the paper in the future.

Reviewer_79PeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes finite-support, multiset, and empirical-measure representations. It formalizes duplicate consistency and geometric stability, then audits maximum regression, pretrained point-cloud classifiers, and trajectory predictors under alternative registrations of the same underlying datum.

Critical strengths
  1. The formal distinctions and assumptions are precise. The factorization and topology arguments clearly separate exact consistency from geometric stability.
  2. The point-cloud audit includes shared inputs, native checkpoint validation, geometric coverage controls, and paired object-level uncertainty. Limitations are carefully disclosed.
Critical weaknesses
  1. The operational advance is limited. Theorem 3.2 directly characterizes support invariance, while Proposition 4.1 assumes the required continuity modulus. Neither provides a checkable certificate for the evaluated learned models, and the study does not demonstrate a model-selection or training improvement enabled by the framework.
  2. The real-task measurements only partially test the central criterion. Point-cloud resampling changes finite supports, so it does not isolate duplicate inconsistency from geometric sensitivity. Trajectory evaluation measures selected ADE/FDE changes and freezes AgentFormer's DLow proposals (Sections 5.2-5.3), leaving prediction-level and full-pipeline consistency unresolved. These acknowledged restrictions narrow the empirical conclusions.
Questions for authors
  1. Can a framework-guided intervention reduce duplicate sensitivity without sacrificing task performance, relative to simple duplicate augmentation?
  2. Can the audits include exact fixed-support point-cloud duplication and direct forecast displacement with native context-dependent proposals?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 79Pe (1/3): Framework consequences and architectural guidance

Thank you for recognizing the precise assumptions and controlled audit design.

Neither provides a checkable certificate for the evaluated learned models

The paper provides structural sufficient conditions, but does not estimate global Lipschitz constants for the complete released checkpoints; D.6 states this limitation. A reproducible nonzero exact-support defect can witness inconsistency, whereas finitely many small defects cannot certify universal consistency. We will keep the distinction between architectural guarantees and empirical checkpoint evidence explicit.

the study does not demonstrate a model-selection or training improvement enabled by the framework.

The intended contribution is the identification and analysis of a representation problem that conventional benchmark evaluation can leave hidden. The paper already gives architectural guidance through Proposition D.1 and the stability discussion in D.6–D.9. Demonstrating a new training method is a separate contribution and lies outside the scope of the present paper.

Can a framework-guided intervention reduce duplicate sensitivity without sacrificing task performance, relative to simple duplicate augmentation?

We are not proposing an augmentation method, and do not plan that training comparison in this revision. Max-based constructions already provide a sufficient consistency condition under Proposition D.1's assumptions.

More fundamentally, the geometric perspective changes what task performance should assess. In ModelNet40, the object being classified is an underlying CAD shape; a uniformly sampled 1,024-point cloud is one finite observation of it. Restricting both training and evaluation to that acquisition protocol creates a benchmark bias toward performance on those registrations. Models can specialize to the protocol while remaining unreliable when the same geometry is registered differently.

Therefore, we evaluate predictive performance together with representation stability across controlled acquisitions. The paper's claim is that native benchmark performance alone misses this requirement, not that a new intervention has already outperformed duplicate augmentation.

AuthorsOpenReview ↗

Response to Reviewer 79Pe (2/3): Geometric objects and sampling bias

Point-cloud resampling changes finite supports, so it does not isolate duplicate inconsistency from geometric sensitivity.

We will shift the emphasis from exact idempotence to near-idempotence and geometric set data representation. Experiment 2 starts from an underlying object and treats each cloud as a finite approximation. Different samples of the same surface generally have different supports; geometric stability across those samples is the intended question. Appendix E.1 already identifies the CAD object and support-level near-idempotence; we will foreground this in §5.2.

Fixed cardinality and uniform sampling impose a particular acquisition law. Training and evaluating only under that law can favor specialization to the sampling protocol while concealing instability when the same object is acquired differently. This is the benchmark bias our audit addresses.

The 256 fixed FPS anchors CC give the support bound

\implies d_H(S,A_m)\le d_H(S,A_n)\le d_H(S,C).$$ For nonnested $A,B\subseteq S$ containing $C$, $d_H(A,B)\le d_H(S,C)$. The CAD-surface and released-point source populations are evaluated separately in a common frame within each comparison. There is also a measure analogue for the CAD surface. Let $\mu=\sigma_S/\sigma_S(S)$ be normalized surface area and $$\nu_B=\sum_{b\in B}\mu(V_b^B)\delta_b,$$ where $V_b^B$ are surface Voronoi cells, with ties assigned measurably. For $1\le p<\infty$, nearest-point transport gives $$W_p(\mu,\nu_B)^p=\int_S d(x,B)^p\\,d\mu(x).$$ Consequently, $$C\subseteq A_n\subseteq A_m\subseteq S \implies W_p(\mu,\nu_{A_m})\le W_p(\mu,\nu_{A_n}) \le W_p(\mu,\nu_C)\le d_H(S,C).$$ Using unnormalized area weights preserves total area $M=\sigma_S(S)$; under the unnormalized equal-mass transport-cost convention, the final bound becomes $M^{1/p}d_H(S,C)$. This is a nesting of approximation errors, with Voronoi weights recomputed on each support. It does not require the Voronoi partitions themselves to be nested. For nonnested $A,B$ sharing $C$, the triangle inequality gives $W_p(\nu_A,\nu_B)\le2d_H(S,C)$. Thus, coverage controls both support approximation and an explicitly surface-weighted approximation. The measure statement specifies the required geometric weights; it is not a bound for arbitrary uniform observation weights or a claim that the audited checkpoints use these weights. We will include this argument in the revision. The observed architectural pattern is relevant. D.2 gives a Hausdorff bound for the PointNet-style pointwise-max construction; PointNet++ uses local PointNet blocks and metric neighborhoods, subject to the selection conditions in D.6–D.7; PointConv compensates for density, with a conditional analysis in D.8. Their small centered-logit changes are consistent with these mechanisms. > Can the audits include exact fixed-support point-cloud duplication Experiment 1 already isolates exact fixed-support multiplicity changes. It can be interpreted as function approximation on one-dimensional point clouds $A\subset[0,100]$, with target $F(A)=\max A$. Its registration-fiber audit changes multiplicities while keeping the support fixed, and the target satisfies $|\max A-\max B|\le d_H(A,B)$. We will make this geometric interpretation explicit. Experiment 1 provides the controlled exact-consistency audit; Experiment 2 addresses near-idempotence across approximations of an underlying object. We will retain this division and will not add the requested exact-support test for the seven point-cloud checkpoints. The claims for those checkpoints concern the measured geometric sensitivity.
AuthorsOpenReview ↗

Response to Reviewer 79Pe (3/3): Direct forecasts with native context-dependent proposals

Can the audits include [...] direct forecast displacement with native context-dependent proposals?

This additional evaluation is complete for all three trajectory models. AgentFormer recomputes its context encoding, native DLow proposals, and decoded forecasts on the corrupted context using the existing checkpoints.

The paired comparisons use:

  • Social GAN: the 20 recorded scene-level latent draws, with the generator rerun on the changed context.
  • Social-STGCNN: the same standard-normal draws for each original pedestrian, forecast, and time step, with graph features and Gaussian parameters recomputed.
  • AgentFormer: deterministic proposal indices in native DLow mean=True\texttt{mean=True} inference.

For each original pedestrian-window, we average Euclidean displacement over all K=20K=20 forecasts and T=12T=12 prediction steps, then average within each fold and equally over the five folds: Dpred=15∑d=151NdKT∑i,j,t∥y^d,i,j,t,1−y^d,i,j,t,0∥2.D_{\mathrm{pred}}=\frac{1}{5}\sum_{d=1}^{5}\frac{1}{N_dKT}\sum_{i,j,t}\left\lVert\widehat{y}^{\\,1}_{d,i,j,t}-\widehat{y}^{\\,0}_{d,i,j,t}\right\rVert_2. Each condition includes 33,654 original pedestrian-windows. Inserted records are excluded from scoring, and no best-of-20 selection enters the statistic.

Mean forecast displacement in metres:

Added records kkσ\sigma (m)Social GANSocial-STGCNNAgentFormer
100.0000001620.17410.0604
10.020.00570.17760.0616
10.050.01270.17350.0672
200.0000003120.21350.0899
20.020.00920.20650.0908
20.050.02110.20030.0956
400.0000004260.25280.1285
40.020.01300.21800.1291
40.050.03010.21090.1331

Trajectory evaluation measures selected ADE/FDE changes and freezes AgentFormer's DLow proposals (Sections 5.2-5.3), leaving prediction-level and full-pipeline consistency unresolved.

AgentFormer uses recomputed proposals throughout the table. With four added records, displacement at σ=0.02\sigma=0.02 is 0.1291 m, compared with 0.1376 m under frozen proposals. At σ=0.05\sigma=0.05, the corresponding values are 0.1331 and 0.1396 m.

Thus, the measured forecast sensitivity persists when the context-dependent proposal step is included. These results address the prediction-level question under the stated couplings and complement the submitted conditional analysis.

The additional Social-STGCNN sweep separates the original source A from the other original pedestrians. For four added records, using the same paired standard-normal draws:

σ\sigma (m)Original A displacement (m)Other originals displacement (m)
00.34780.2139
0.00010.28400.2715
0.020.26320.2080
0.050.25780.1966

Both statistics average all 20 forecasts and 12 future steps, followed by equal averaging over the five folds. Within each fold, the source statistic averages one A per scene; the other statistic pools the remaining original pedestrian-windows. They complement the all-original-pedestrian table above. Inserted copies are excluded throughout.

The source A and duplicate-noise directions are fixed across conditions; σ\sigma denotes positional-noise standard deviation per coordinate and observed step.

Reviewer_aNXxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper observes that permutation invariance removes order but not multiplicity, so a variable-cardinality family Fn:Xn→YF_n: X^n \to Y may represent a set, a multiset, or an empirical measure depending on which changes of "registration" are meant to preserve the datum. It defines idempotent set consistency (Fn=f∘snF_n = f\circ s_n for a function ff on finite supports), proves that this holds iff FF is permutation invariant and compatible with duplicate insertion (Theorem 3.2), counts the resulting registration fibers (Prop. A.2), defines quantitative fiber defects (Eq. 2, A.4), gives an equivariant version for per-element outputs (Prop. A.4), and separates exact consistency from Hausdorff-geometric stability (Prop. 4.1; Appendix C counterexamples showing the direct-limit topology is finer than the Hausdorff/Wasserstein metric topology). Three audits are reported: (1) maximum regression (Lee et al. 2019) with mean/sum/max/log-mean-exp pooling and a Set Transformer, exhaustively enumerating all length-100 count vectors on 2- and 3-element supports via count-compressed attention (Prop. D.2); (2) seven frozen ModelNet40 classifiers evaluated on two controlled source populations under density interventions (λ∈{0,1,2,4}\lambda\in\{0,1,2,4\}), coverage control (256 FPS anchors), repeat registrations and nested 512/1024/2048 cardinality stress with fixed internal hierarchy counts, reporting a relative centered-logit change RR with class-stratified object bootstrap intervals; (3) duplicate and near-duplicate track insertion for Social GAN, Social-STGCNN and AgentFormer on ETH/UCY under fixed inference couplings, reporting macro-averaged absolute ADE/FDE changes. Main empirical claims: Set Transformer learns near-idempotent behaviour without a structural guarantee; DGCNN, PointMLP, PointMamba and PointGPT-S show large representation changes under density/cardinality interventions while PointNet++/PointConv do not; AgentFormer and Social-STGCNN errors change by 15–40 % of baseline under 2–4 duplicate records while Social GAN is unchanged.

Critical strengths

S1 — Rigorous, paired, reproducible audit protocol (§5.2, Appendices F, G, D.5). Shared stored input banks across seven checkpoints and two source populations, class-stratified object-cluster bootstrap on paired differences (G.8), numerical-determinism gates (G.6, G.9), exhaustive fiber enumeration made tractable by the count-compression equivalence (Prop. D.2), and complete per-seed tables (Tables 7–8). This is consequential because it makes the empirical statements trustworthy and the audit directly reusable by others; it is well above the usual standard for sensitivity studies.

S2 — A clean, measurable formalization plus one genuinely informative finding (§3, Prop. A.4, Table 1). Graded registration fibers and the defect ΔF(K)\Delta_F(K) turn the informal "sets vs. multisets" distinction into quantities that can be reported per model, and the equivariant descent statement (Prop. A.4) is the right object for per-agent outputs. The Set Transformer result (three-support mean defect 0.004 and diameter 0.034 vs. 3.4/44.9 for mean pooling, with no structural guarantee and no duplicate augmentation, F.2) is a concrete data point about what attention learns on this task.

Critical weaknesses

W1 — The conceptual framework is a formalization of a known distinction and is not load-bearing [Fundamental to the stated contribution; partially fixable by repositioning].

  • Issue and location: §1–3 and Appendices A–C present "registration consistency" as a new representation-level framework. The trichotomy max ↔ set, mean ↔ distribution, sum ↔ multiset is stated explicitly in Xu et al. (ICLR 2019, Figure 2 and §5.2, "max-pooling learns sets with distinct elements"), which the paper does not cite, and the idempotence-of-max versus multiplicity-sensitivity-of-sum point is in Portilheiro (2019), which it does cite. Theorem 3.2's "iff" has a three-line proof; Prop. A.2 is the count of surjections and of compositions; the Appendix C counterexamples are the standard fact that a filtered colimit's final topology is finer than the metric topology (Example C.4 is the dyadic rationals), which the paper itself calls an "analogy".
  • Gap: No result in the paper depends on more than Definition 3.1: no bound, impossibility statement, certificate for a trained pipeline, or training-time remedy is derived from Thm 3.2 or Appendix C, and the audits would be unchanged without them.
  • Impact: The claimed "framework" contribution reduces to a checklist (B.5); what remains is three descriptive audits, which is below the bar for a main-track paper unless the audits themselves are decisive (see W2).
  • Suggested remedy: Reposition as an empirical audit paper; cite and explicitly differentiate from Xu et al. 2019 and Portilheiro 2019; compress Appendices B–C to a remark; or add one result that only the framework can give (e.g. a consistency regularizer or architecture change evaluated against the fiber-defect metrics, or a certified bound for one audited pipeline).

W2 — The point-cloud and trajectory audits cannot distinguish semantic registration inconsistency from ordinary distribution shift, and they lack the baselines needed to interpret their numbers [Fixable].

  • Issue and location: (a) Exp. 2's headline effects come from λ=4\lambda=4 (sliced-W2W_2 0.259 vs. 0.019 for repeat pairs, §5.2) and from feeding 512/2048-point clouds to checkpoints trained at 1024 with frozen hierarchy counts (G.6, "explicit stress protocol"). The paper states that support and surface-measure semantics are "complementary rather than exclusive" for CAD objects (§5.2, E.1) and never says which one a ModelNet40 classifier should satisfy, so Rden4=1.035R^4_{\rm den}=1.035 and 36 % accuracy for DGCNN has no normative reading. ModelNet40-C (Sun et al., ICLR 2022), which measures exactly Local_Density_Inc/Dec, Upsampling and Cutout corruptions on the same 2,468 objects for PointNet, DGCNN and PointNet++, is neither cited nor compared against, so it is unclear what the audit reveals beyond known corruption sensitivity. The direct test of Definition 3.1 for point clouds, exact duplicate insertion at the native n=1024n=1024, is not run. (b) Exp. 3's AgentFormer result holds only along frozen DLow proposals (I.7), the metric A=⟨∣ΔM∣⟩A=\langle|\Delta M|\rangle satisfies A≥∣S∣A\ge|S| and cannot distinguish degradation from improvement (I.10), and the Social GAN null is explained by the max-pooling identity (I.8). Related perturbation studies of social context (Saadatnejad et al. 2022; CausalAgents, Roelofs et al. 2022) are not discussed.
  • Gap: The inference "accuracy persists but representation changes ⇒ the model does not respect the intended semantics" requires (i) a stated intended semantics and (ii) a control that is within the model's training contract; both are missing, so the conclusion collapses to "outputs change under input perturbation", which is true of any model.
  • Impact: This is the empirical support for the paper's central claim (abstract: the framework "tests whether a model respects the one intended by the application"); as run, it does not test that.
  • Suggested remedy: Add an exact-duplicate and exact-resample control at n=1024n=1024 for all seven checkpoints; compare RR and accuracy against ModelNet40-C density corruptions at matched Hausdorff/sliced-W2W_2 severity; state per application which semantics is intended and report whether each model satisfies it; report signed changes and a recomputed-proposal variant for AgentFormer.
Questions for authors
  1. (Clarification; drives W1) Beyond formalization, what does the framework add to Xu et al. 2019 (Fig. 2, §5.2) and Portilheiro 2019? Is there any statement in the paper (bound, impossibility, certificate, or method) that follows from Theorem 3.2 or Appendix C and could not be made from those works plus Definition 3.1?
  2. (Clarification; drives W2) For the ModelNet40 classifiers, which semantics is intended under the λ\lambda intervention and under 512→2048 refinement: should the representation be invariant, and if so to what tolerance? What does Rden4R^4_{\rm den} for DGCNN show that ModelNet40-C's Local_Density corruptions did not?
  3. (Clarification; drives W2) Exact duplicate insertion at the native cardinality is the within-contract test of Definition 3.1. Do the supplementary results contain it, and what are the per-checkpoint defects?
  4. (Clarification; drives W2) For AgentFormer and Social-STGCNN, what are the signed changes SS (Eq. 54) for k=2,4k=2,4, and how do the results change when DLow proposals are recomputed on the corrupted context?
  5. (Clarification; drives S2/W1) Does Set Transformer's near-idempotence persist for supports of size 5–10, registration length 1000, and for other support-factoring targets (e.g. median or range), or is it specific to max with two attention blocks?
12 comments
AuthorsOpenReview ↗

Response to Reviewer aNXx (1/4): Prior work and the role of the framework

Thank you for recognizing the rigor of the audit protocol and the informative Set Transformer result. The revision will shift the main emphasis from exact idempotence to near-idempotence and geometric set data representation. Exact compatibility remains the foundation; stability under geometric approximation is the central representation requirement.

The trichotomy max ↔ set, mean ↔ distribution, sum ↔ multiset is stated explicitly in Xu et al. [...] which the paper does not cite

We agree that this aggregator distinction is established and will add the omitted Xu et al. (2019) citation. Portilheiro (2019) is already cited.

Xu et al.'s Corollary 9 establishes support discrimination by max aggregation on a countable feature domain. It does not characterize geometric proximity between registrations, a continuity modulus for nearby supports, or extension toward an underlying compact object.

Our contribution concerns geometric set data representation. The top-down perspective starts with an underlying object and its geometry; finite registrations are observations or approximations of that object. The directed-system construction then organizes those registrations and their compatibility. For a CAD surface, both its compact support and geometric area measure remain fixed when acquisition density changes, although the empirical observation measure changes. This distinction determines which approximation and consistency requirements the representation should satisfy. We will foreground this perspective, including the limits connecting finite observations to continuum objects, when differentiating our work from the aggregator trichotomy.

Theorem 3.2's "iff" has a three-line proof; Prop. A.2 is the count of surjections and of compositions

Theorem 3.2 gives necessary and sufficient compatibility conditions for an arbitrary variable-cardinality family, independently of a pooling rule. Its short proof does not remove the role of that characterization. Proposition A.2 uses classical counting identities to describe the graded fibers on which defects are evaluated; novelty is not claimed for the identities themselves.

No result in the paper depends on more than Definition 3.1 [...] and the audits would be unchanged without them.

The framework determines which comparisons are meaningful and what their defects establish. Theorem 3.2 characterizes exact support factorization; Proposition A.4 gives its per-element counterpart; Appendix C separates registration topology from the geometric topology needed for approximation. These distinctions guide the interpretation of the experiments even though the numerical statistics can be computed without invoking every theorem.

We will make the connection to a quantitative consequence explicit. Let X,YX,Y be complete metric spaces, and suppose a permutation- and duplicate-compatible family satisfies a common bound dY(fn(x),fn(y))≤Lmax⁡idX(xi,yi)d_Y(f_n(x),f_n(y))\le L\max_i d_X(x_i,y_i) for every cardinality. Two finite supports admit common-length registrations whose maximum paired distance equals their Hausdorff distance. Compatibility then gives an LL-Lipschitz support map, which extends uniquely to nonempty compact subsets of XX. Consequently, dY(F(A∪z),F(A))≤L,dX(z,A).d_Y(F(A\cup\\{z\\}),F(A))\le L\\,d_X(z,A). This identifies the additional regularity needed to pass from exact compatibility to near-idempotence. Proposition D.1 provides a concrete architectural instance. The statement synthesizes compatibility and geometric regularity; it does not make the classical max bound a new result.

add one result that only the framework can give (e.g. a consistency regularizer or architecture change [...] or a certified bound for one audited pipeline)

We disagree that demonstrating a new training remedy is necessary to establish the contribution identified here. The framework exposes a bias in what the benchmark measures: a uniformly sampled, fixed-cardinality registration is treated as the datum even when the task concerns an underlying geometric object. A training improvement measured only under that acquisition law would leave this problem unresolved.

The operational consequence is to evaluate representation stability across geometrically controlled acquisitions alongside native task accuracy. The audited checkpoints show why the distinction matters: strong performance under the benchmark protocol does not establish reliability across observations of the same object. We will make this consequence, the compatibility-to-stability implication above, and the existing architectural analysis in §6 and D.2, D.6–D.9 explicit. A new regularizer or architectural intervention would extend this contribution but lies outside the scope of the present paper. Neither is required to establish the acquisition dependence demonstrated by the audit.

The new Social-STGCNN diagnostic below also identifies a collision discontinuity in graph construction, providing an architectural diagnosis before training.

AuthorsOpenReview ↗

Response to Reviewer aNXx (2/4): Topology, near-idempotence, and other targets

the Appendix C counterexamples are the standard fact that a filtered colimit's final topology is finer than the metric topology

Could you identify the specific theorem, proposition, or corollary supporting this statement, with its number or other precise location and relevant hypotheses, and explain which Appendix C result or construction it covers? Please cite the particular result in the literature; a reference to a paper or textbook as a whole would not identify the asserted overlap.

The general comparison obtained when stage maps into a metric space are continuous should be distinguished from strict separation of the relevant topologies and from the metric completion needed to represent continuum objects. If the objection concerns one of these particular statements, an explicit theorem or reference would help us address the claimed overlap precisely.

Appendix C examines whether the topology supplied by the registration construction agrees with the geometric notion of approximation required by the application. Its role is to clarify the passage from finite registrations to underlying objects. We would appreciate the reviewer's specifying which part is covered by the asserted standard result.

Does Set Transformer's near-idempotence persist for supports of size 5–10, registration length 1000, and for other support-factoring targets (e.g. median or range)

Experiment 1 can already be interpreted as function approximation on one-dimensional point clouds A⊂[0,100]A\subset[0,100], with target F(A)=max⁡AF(A)=\max A. Increasing the support size to 5–10 or the registration length to 1,000 extends the cardinality and multiplicity stress within this same geometric learning problem. The ambient space, target, and intended consistency criterion remain the same. In particular,

∣max⁡A−max⁡B∣≤dH(A,B)|\max A-\max B|\le d_H(A,B)

holds independently of support size and registration length.

An exhaustive extension of the submitted registration-fiber audit is computationally prohibitive. The main text already gives the count (n−1r−1)\binom{n-1}{r-1} for length-nn registrations on a fixed rr-element support, modulo permutations. Taking n=1000n=1000 and r=mr=m gives (999m−1).\binom{999}{m-1}. This is approximately 4.13×10104.13\times10^{10} vectors for m=5m=5 and 2.63×10212.63\times10^{21} for m=10m=10, per fixed support. Sampling registrations remains feasible, but it would replace exhaustive fiber coverage with a sampled audit.

These settings would provide additional evidence about how the trained Set Transformer scales. They do not introduce a new geometric requirement. We will make the point-cloud interpretation of Experiment 1 explicit and retain the empirical consistency claim within the reported settings; the target's stability alone does not establish the trained model's behavior at untested sizes.

Range is a suitable target for the geometric extension because ∣range⁡(A)−range⁡(B)∣≤2dH(A,B).|\operatorname{range}(A)-\operatorname{range}(B)|\le2d_H(A,B). Median requires a distinction. The ordinary registration median is multiplicity-sensitive. Even the median of distinct support points can fail Hausdorff continuity when points collide: for 0<ε<1/30<\varepsilon<1/3, Aε={0,ε,2ε,1,2}⟶H{0,1,2}.A_{\varepsilon}=\lbrace 0,\varepsilon,2\varepsilon,1,2\rbrace\longrightarrow_H\lbrace 0,1,2\rbrace. but med⁡(Aε)=2ε→0\operatorname{med}(A_\varepsilon)=2\varepsilon\to0, whereas the limiting support has median 1.

This does not prevent learning median on restricted domains. It explains why range and median do not test the same geometric property. No additional range or median results are claimed in this response.

AuthorsOpenReview ↗

Response to Reviewer aNXx (3/4): Geometric semantics and benchmark bias

support and surface-measure semantics are "complementary rather than exclusive" [...] and [the paper] never says which one a ModelNet40 classifier should satisfy

The revision will foreground near-idempotence: geometric stability across finite approximations of the same underlying object. Appendix E.1 already fixes the CAD surface SS and its area measure σS\sigma_S under acquisition-density changes, and identifies support-level near-idempotence as Experiment 2's primary criterion. The empirical sampling measure may change without changing the geometric object or class. "Complementary" refers to these two geometric descriptions of that object.

Fixed 256 FPS anchors CC gives

\implies d_H(S,A_m)\le d_H(S,A_n)\le d_H(S,C).$$ For nonnested $A,B\subseteq S$ sharing $C$, $d_H(A,B)\le d_H(S,C)$. The measure analogue uses $\mu=\sigma_S/\sigma_S(S)$ and Voronoi weights $$\nu_B=\sum_{b\in B}\mu(V_b^B)\delta_b.$$ For $1\le p<\infty$, nearest-point transport gives $$W_p(\mu,\nu_B)^p=\int_S d(x,B)^p d\mu(x),$$ so the same nested supports satisfy $$W_p(\mu,\nu_{A_m})\le W_p(\mu,\nu_{A_n}) \le W_p(\mu,\nu_C)\le d_H(S,C).$$ Unnormalized area weights preserve total area $M=\sigma_S(S)$; the corresponding unnormalized transport bound is $M^{1/p}d_H(S,C)$. Weights are recomputed on each support, with measurable tie breaking. This bounds surface-weighted approximation errors; it does not assert nested Voronoi cells or the same bound for uniform observation weights. We will add this geometric measure argument; the audited checkpoints were not supplied with these Voronoi weights. > $R_{\mathrm{den}}^4=1.035$ and 36% accuracy for DGCNN has no normative reading The object and class remain fixed, providing the task-level expectation of reliable recognition across informative acquisitions. Accuracy loss has a direct task meaning. The centered-logit ratio measures representation sensitivity; it is not itself a universal continuity threshold. Our contention is that fixed-cardinality, uniformly sampled evaluation is biased toward one acquisition law. It can reward specialization to those registrations while concealing dependence on how the object is sampled. The large responses of DGCNN, PointMLP, PointMamba, and PointGPT-S expose such acquisition dependence. D.2 supplies the PointNet-style pointwise-max Hausdorff bound. PointNet++ uses local PointNet blocks and metric neighborhoods, with the selection conditions in D.6–D.7; PointConv compensates for density, with a conditional analysis in D.8. Their comparatively small centered-logit changes are consistent with these mechanisms. The full hierarchical and density-corrected pipelines require the stated conditions. > [The inference] requires [...] a control that is within the model's training contract We report native-length density interventions separately from 512/1,024/2,048-point stress tests. However, the training acquisition law should not define the entire object-recognition task. Testing geometrically controlled acquisitions of the same surface examines precisely the benchmark bias left untested by that law. > ModelNet40-C [...] is neither cited nor compared against We will cite ModelNet40-C, ICLR 2022 Workshops. Its generator does not enforce one fixed center and scale across all the listed corruption types: [upsampling](https://github.com/jiachens/ModelNet40-C/blob/master/data/generate_c.py#L114-L120) normalizes the resulting cloud again, while density and cutout retain the input coordinates. Our protocol fixes the same $(c,s)$ within each object comparison, including independently acquired CAD samples, and fixes coverage anchors. This makes acquisition changes comparable in a common geometric frame. Together with separate source populations, shared inputs across seven checkpoints, and paired uncertainty, these controls address geometric acquisition consistency. We will not add the requested matched-corruption experiment. > Exact duplicate insertion at the native $n=1024$ [...] Do the supplementary results contain it? No; the point-cloud repeat controls use independent samples. Experiment 1 already isolates exact fixed-support multiplicity changes: it learns $F(A)=\max A$ on one-dimensional point clouds $A\subset[0,100]$ and audits registrations of fixed supports. The target is Hausdorff-Lipschitz. We will make that interpretation explicit and retain Experiment 1 for exact consistency. Experiment 2 will emphasize near-idempotence across approximations of underlying objects; we will not add an exact-support test for the seven checkpoints or interpret resampling as that test.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.