acceptodds
Under review as a conference paper at ICLR 2027

GenSymAudit: Anchor-Relative Auditing of Distributional Symmetry in Generative Models

Abstract

Generative models are increasingly deployed through fixed sampling and conditioning pipelines, yet structural properties of their output distributions are often inferred from the training data or ancestor model rather than measured on the deployed artifact itself. We study this problem through distributional symmetry and introduce GenSymAudit, a sample-only framework for auditing whether a deployed generative artifact changes the element-wise symmetry profile of the real distribution it is intended to model. This problem cannot be addressed by testing generated samples against perfect symmetry: real data may already contain substantial directional structure, while the deployed distribution additionally depends on the model checkpoint, conditioning protocol, sampler, guidance, and sampling budget. GenSymAudit constructs condition-matched real and generated pools, measures transformation-specific separability using grouped classifier two-sample tests, and assigns anchor-relative verdicts using negative controls, calibrated uncertainty, multiplicity correction, and quality gates. Both fMoW and Git-10M exhibit strong departures from perfect invariance. Relative to these real-data anchors, DiffusionSat exhibits three statistically resolved amplifications, dominated by vertical reflection (+18.11 points), while Text2Earth shows smaller bidirectional changes, attenuating horizontal reflection (-2.98 points) and amplifying vertical reflection (+3.62 points). Controlled few-step experiments further show that likelihood-derived symmetry certificates can become numerically unreliable while independent sample-space measurements continue to detect asymmetry. These findings support the broader conclusion that structural properties should be measured on the complete artifact actually deployed rather than inferred from its reference data, ancestor model, or nominal sampling configuration.

Primary area:
generative models
Keywords:
Generative Model Auditing, Model Evaluation, Distribution Shift, Symmetry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QDBzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces GenSymAudit, a sample-based framework for auditing whether a deployed generative model preserves, amplifies, or attenuates the symmetry properties already present in real data. For each transformation in a finite group such as rotations and reflections in D4D_4, the method uses classifier two-sample tests to measure how distinguishable the original samples are from their transformed versions, separately for the real-data anchor and the generated distribution. It then subtracts these two asymmetry scores and applies bootstrap confidence intervals, multiple-testing correction, equivalence testing, negative controls, and quality gates to produce transformation-specific verdicts. Experiments on DiffusionSat and Text2Earth show that some asymmetries are inherited from their respective real datasets, while others are selectively amplified or attenuated by the deployed generators. The paper argues that anchor-relative auditing provides a more informative assessment than simply testing whether generated data are perfectly symmetric.

Critical strengths

A key strength of this paper is its careful and systematic statistical evaluation. By comparing generated data against a matched real-data anchor, the framework can distinguish asymmetry inherited from real data from changes associated with the generator. Its element-wise analysis, together with bootstrap uncertainty, negative controls, multiple-testing correction, and equivalence testing, makes the resulting audit more reliable and interpretable than a simple test of absolute symmetry.

Critical weaknesses

The main weakness of this paper is its limited novelty and significance. It mainly extends existing symmetry-auditing methods into a more elaborate statistical pipeline, while the underlying problem of whether generative models preserve data symmetry has already been studied [1, 2]. Moreover, similar anchor and generated-data scores indicate only comparable levels of separability, not necessarily preservation of the same underlying symmetry structure. Although the paper includes downstream experiments, the observed performance gaps may arise from many differences between real and generated data, and the reported associations with symmetry shifts remain exploratory and inconsistent across datasets. The paper therefore does not establish a causal link between the detected distortions and downstream harm, nor does it propose a method for correcting them.

  1. Caytuiro, Nicolas, and Ivan Sipiran. "Symmetry matters: Auditing and symmetrizing 3D generative models." Computers & Graphics (2026): 104687.
  2. Lawrence, Hannah, et al. "To augment or not to augment? diagnosing distributional symmetry breaking." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors

How can the authors demonstrate that similar symmetry scores for the anchor and generated distributions indicate preservation of the same underlying symmetry structure, rather than different asymmetries of similar magnitude?

What is the key novelty of GenSymAudit compared with existing symmetry-auditing methods and a simple subtraction of two C2ST-based symmetry scores?

Can the authors causally isolate the effect of the detected symmetry deviations on downstream performance, for example through a controlled intervention that changes symmetry while preserving other aspects of generated-data quality?

Reviewer_rTbcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a deployed generative artifact, meaning not only the model but also its conditioning, sampling, guidance, and inference budget, preserves the structural properties of its reference distribution. To this end, the authors propose GenSymAudit, a sample-only framework that audits whether a deployed artifact changes the element-wise symmetry profile of the real distribution it is meant to model, using real-data anchors rather than perfect symmetry as the reference. They evaluate it on two remote-sensing generators. For DiffusionSat, the fMoW reference distribution is itself strongly asymmetric under D4, and relative to this anchor the generator shows three resolved amplifications, dominated by vertical reflection. For Text2Earth, the changes relative to Git-10M are smaller and bidirectional: horizontal reflection is attenuated and vertical reflection is amplified.

Critical strengths

A well-motivated problem since real data are rarely symmetric, so testing generated samples against perfect invariance is the wrong target. Auditing the deployed artifact relative to a real-data anchor is a sensible and clearly articulated framing.

The negative controls, grouped and stratified splits, power calibration, multiplicity correction, and equivalence testing are more rigorous than is typical in generative-model evaluation. The negative controls exposed real leakage (0.5477 under per-image splits, down to ~0.50 after grouping and stratification), and the authors openly report a metadata-alignment bug found during the Text2Earth screen.

The paper distinguishes exploratory from confirmatory claims and acknowledges classifier dependence, headroom effects, and the post-hoc subgroup analysis.

The DiffusionSat vertical-flip shift is large and holds across classifier capacities

Critical weaknesses

The contribution has very limited novelty, it is a difference of two classifier two-sample test scores, wrapped in standard statistical machinery. Using C2ST to detect symmetry breaking is established (Lopez-Paz & Oquab, 2017; Lawrence et al., 2026), and “compare to the real data’s own baseline” is already how generative evaluation works. There is no new estimator, no theory beyond the textbook C2ST/TV relation, and no new model. This is a useful protocol, but not a machine learning paper.

The measurement does not support the interpretation. C2ST accuracy from a CNN trained for three epochs per fold is capacity- and optimisation-dependent, as the authors concede, yet verdicts are phrased as statements about distributional symmetry. Differences of saturating accuracies are hard to interpret when anchors differ. Appendix B admits that the axis-exchanging transformations sit near 0.83 on fMoW. The classifier may also be picking up low-level, orientation-dependent image-formation differences (decoder or upsampling structure, bilinear resizing of real images versus native generation). No control isolates these, such as passing real images through the generator’s autoencoder or matching the resizing pipeline.

The DiffusionSat arm generated draws share the first acquisition’s conditioning, while the real pool contains later acquisitions. Guidance scale is fixed at 7.5 and never varied. This undercuts the paper’s thesis that the whole artifact matters, since only step count is varied. For Text2Earth, the anchor comes from the generator’s own training distribution, with possible tile overlap acknowledged by the authors.

The verdict logic is internally inconsistent. The equivalence margin is δ = 0.05, yet three of the five “resolved” shifts are smaller than δ (+4.39, −2.98, +3.62 points). Appendix A.3 acknowledges this, which means a shift can be both equivalent to the anchor within the margin and “amplified”, resolved only by the order in which the tests are applied. δ is itself set to the coarsest demonstrated detection resolution, so it is data-driven rather than tied to practical importance, and the equivalence verdicts change under modest changes to it (four DiffusionSat equivalences become two at δ = 0.03). The “invented” category defined in Sec. 3 is never reported.

No baselines, and narrow scope relative to the claims. The paper never shows that existing tools fail to detect the same effects, so the need for a new framework is not demonstrated. The empirical base is two remote-sensing generators and one group. The only natural-image experiment (ImageNet-64 consistency models, three seeds) is a null result in five of six arms. The broad conclusion that symmetry must be audited on the deployed artifact rather than inferred from data, ancestry or sampling configuration goes well beyond this evidence.

Questions for authors
  1. Can you show the DiffusionSat vertical-flip shift is not caused by orientation-dependent image-formation differences? How do the three DiffusionSat amplifications change across classifier-free guidance scales (e.g., 1, 3, 7.5, 12), and when generated draws use the metadata of the matching real acquisition instead of only the first one? Since the paper argues the whole artifact matters, why is step count the only factor varied? The Appendix B says the concentration of shifts in V may be confounded by discriminator headroom, and B.3 shows weaker cells moving across classifiers. Do the verdicts, especially the 3-4 point Text2Earth shifts and the V-only pattern, survive (a) a saturation-robust statistic such as a log-odds transform, (b) a non-learned test on the same pools, and (c) a different architecture with longer training?

Three of the five “resolved” shifts are smaller than δ = 0.05. How should a reader interpret a shift that is both significant and within the equivalence margin? Why not use a single rule based on where the confidence interval falls relative to ±δ (outside, inside, straddling), with δ set from a practical criterion rather than the detection resolution?

Do existing tools (feature-space distances, precision/recall-type metrics, the kernel invariance test of Soleymani et al., 2025) detect the same effects on the flipped versus unflipped pools? And can you apply the audit to a widely used text-to-image model with default guidance on natural images, to support the general claim?

Reviewer_dW3GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper examines the following question: does a deployed image generator change the symmetry profile of its training data under D4, the 7 non-identity rotations and flips of the square pixel grid? A “deployed artifact” in the context of this paper is the full pipeline: weights plus sampler, step count, guidance and conditioning.

Real data is not D4-invariant: shadows, north-up orientation and acquisition geometry break the symmetry (indeed, finding the correct orientation of a natural image is one of the first proposed self-supervised pretext task strong enough to pre-train an image classifier). Testing whether generated images are perfectly invariant is therefore the wrong question. The authors measure asymmetry relative to the real data, the “anchor” (eq. (1) and (2) in the paper). These two equations are equivalent to:

Δg=S^g(QA)−S^g(P),\Delta_g = \hat S_g(Q_{\mathcal A}) - \hat S_g(P),

where S^g\hat S_g is the held-out accuracy of a classifier two-sample test (C2ST) separating xx from g⋅xg·x. The real images P and the generated images Q_A are each tested this way, separately.

Some safeguards are implemented:

  • grouped splits, so images of the same site never straddle train and test;
  • negative controls;
  • power calibration with injected asymmetries;
  • Holm correction for multiple tests;
  • equivalence testing with margin δ = 0.05;
  • a gate that discards outputs whose quality has degraded.

The paper presents the following results:

  • DiffusionSat amplifies the asymmetry of its fMoW training data under vertical flip (+18 points of C2ST accuracy, 57% → 75%), horizontal flip (+6) and 180° rotation (+4);
  • Text2Earth weakens horizontal-flip asymmetry (−3) and strengthens vertical-flip asymmetry (+3.6);
  • All five significant shifts fall in the axis-preserving Klein subgroup {e, r₁₈₀, flip_h, flip_v}.
Critical strengths
  • The core point is right and well argued: compare against the real data, not against perfect invariance. Fig. 3a shows that an absolute test would flag every D₄ element for both generators.

  • The statistical framework is unusually good. The negative-control sequence is a clean demonstration of leakage: 0.5477 with per-image splits, 0.5085 when grouped by site, 0.5002 when also stratified. The authors correctly use equivalence testing instead of reading “not significant” as “preserved”, calibrate power, correct for multiple tests, and check sensitivity to the classifier and to resolution.

  • Limitations are stated candidly: no causal claims, the ceiling effect, and the post-hoc subgroup analysis.

Critical weaknesses
  1. A conditioning mismatch may explain the main result. Appendix A.1 admits that every generated image for a site uses the metadata of that site’s first acquisition, including date and time, which fix the sun position. The real anchor, by contrast, contains up to four acquisitions per site, each with its own metadata. As a consequence, the generated pool has less variation in illumination, so its shadow directions are more concentrated. That alone would increase reflection asymmetry, and vertical flip, the headline +18, is exactly the transformation that moves the dominant south-east shadows. The authors disclose the issue but never quantify it.

  2. DiffusionSat and Text2Earth both run at the same 7.5 classifier-free guidance. Strong guidance is known to sharpen samples toward the modes of the distribution, which would amplify the most common shadow direction. They sweep the number of steps (Fig. 6), not the guidance scale. A guidance sweep is the most natural missing experiment.

  3. Δ_g compares two separability scores, so a generator could reverse the asymmetry and still score Δ = 0: real shadows pointing south-east, generated ones north-west, both equally separable from their flips. The “symmetry profile” does not record direction.

  4. C2ST Accuracy is bounded and nonlinear. The axis-exchanging elements sit at about 0.83 on fMoW, the axis-preserving ones at 0.57–0.68, so equal underlying shifts produce smaller Δ near the ceiling. The authors acknowledge this in App. B.4, but it undermines the localisation to the Klein subgroup, which is the paper’s most interesting observation.

  5. The novelty is somewhat limited:

  • The method combines C2ST, already used for symmetry testing by Lawrence et al. 2026 and Lippmann et al. 2026, with “compare to the real data”, which is standard practice in generative evaluation. The important contribution is really the careful protocol.

  • The case for why symmetry preservation matters is a bit weak. The downstream study shows a large accuracy gap from training on generated data (−42% and −32% relative), but the link to symmetry is exploratory: ρ = 0.82, p = 0.034, uncorrected, on fMoW; nothing on Git-10M. The subgroup p = 0.007 comes from a partition chosen after looking at the results, as the authors admit.

  • The study covers only two generators, one transformation group and one domain.

The use of "Artifact" meaning the deployed pipeline in the context of the paper is non standard and will confuse readers (it confused me).

Questions for authors

1- What happens to the DiffusionSat amplifications, especially vertical flip, when each generated draw is conditioned on its own acquisition’s metadata, or when the anchor is restricted to first acquisitions? 2- How does the profile change with the guidance scale (e.g. 1, 3, 7.5)? 3- Can the audit tell preserved from reversed asymmetry, for example by applying the anchor-trained x-vs-g·x classifier to generated samples? 4- Given the ceiling effects, how do the verdicts change on a calibrated divergence scale instead of raw accuracy, and is the Klein-subgroup localisation still there? 5- What drives the 0.83 rotation separability of real fMoW images: scene layout or sensor anisotropy? Does a feature-space C2ST give the same profile? 6- Beyond the exploratory downstream association, what concrete harm follows from a generator amplifying an existing asymmetry? 7- Please clarify the “historical implementation” and 24-permutation cap, and why the likelihood results appear in the abstract when they come from a toy appendix study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.