acceptodds
Under review as a conference paper at ICLR 2027

Dual-Order Rènyi–Dirichlet Graph Autoencoders for Anomaly Detection on the Simplex

Abstract

Low- and high-dimensional simplex-valued data arise in microbial communities, spatial transcriptomic profiles, particle-size distributions, and normalized image intensities, where observations may be connected by a graph and corrupted by heterogeneous anomalies. We introduce Dual-Order Rényi–Dirichlet Graph Autoencoders (RD-GAEs), a simplex-geometry preserving graph autoencoder that uses Dirichlet latent variables and Rényi divergences at two distinct orders. A subunit training order emphasizes the dominant, regular structure and reduces the influence of contaminated observations during representation learning; after training, a larger detection order amplifies reconstruction discrepancies in distributional tails, yielding anomaly-sensitive scores without retraining the model. The encoder and decoder preserve compositional support through graph aggregation and gated probability transformations, while the graph can be row-stochastic or doubly stochastic. Across controlled-contamination and held-out detection protocols for image, microbiome, and spatial transcriptomics data, RD-GAE achieves superior anomaly-detection performance while retaining interpretable simplex-valued representations, compared with graph, variational, Dirichlet, and Euclidean-coordinate autoencoder baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
anomaly detection, simplex geometry, graph autoencoders, Rényi divergence, robust representation learning, graph neural networks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_bNkEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Dual-Order Rényi–Dirichlet Graph Autoencoders (RD-GAEs), a graph encoder for anomaly detection on simplex-valued data. It learns representation based on the probability simplex of the input. It combines stochastic graph operators, stochastic category mapping, gated probability-temperature map, and Dirichlet bottleneck. The model uses a Rényi order below 1 for training, 0<λtr<10 < \lambda_{tr}< 1 to be exact. After training, it uses an order above one to calculate anomaly scores. The paper gives conditional score-separation results and contamination bounds. Experiments cover two image datasets and two biological datasets.

Critical strengths
  1. The reported improvements are substantial, particularly for AUPRC.
Critical weaknesses
  1. The writing is poor and lacks of clarity. Sections 4.3 and 4.4 repeat the same discussion. Implementation details need more clarification to ensure reproducibility.
  2. The novelty of the proposed method is not well established. The combination of the components can be valuable, but the paper needs to clarify what new insights it provides and how it can be used to solve practical problem. Its practical value also remains uncertain because the evaluation uses only synthetic perturbations.
Questions for authors
  1. Does low-order training still improve performance when a KL-trained model uses the same Rényi detection-order selection procedure?
  2. The main text mentions the detection order is selected by validation AUROC. However, the text below Appendix Table 3 uses AP first, then AUROC to break ties. Which criterion was used in the implementation?
  3. Provide more justification on the results in Figure 4. What are the actual Omniglot contamination rate?
  4. What formula did the implementation use for the softmax gate ablation? Table 6 omits the coefficient gg included in the main text.
  5. Did the softmax-gate ablation use a suitable temperature range?
  6. Can you provide the complete edge-weight formula described in Section 2.1? How do you define the receiver’s local consensus and set the strength of the downweighting?
  7. Theorems 3.1 and 3.3 concern fixed parameters and graph operators. Do you have results that compare low-order and KL training at different contamination rates, with the detection procedure held fixed?
  8. What are the limitations of the proposed methods?
  9. What is the scalability of the RD-GAEs?
  10. Appendix Table 3 states that RD-GAE uses early stopping, while the baselines run to the epoch cap. How were baseline checkpoints selected?
  11. What is ILR-transformated Input?
6 comments
AuthorsOpenReview ↗

Response to Reviewer bNkE (part 1/2)

We thank the reviewer for identifying these gaps and limitations. Following the your suggestions, we clarified the methodology, conducted additional analyses, and revised the main manuscript and Appendix accordingly (see the colored text).

Weakness 1

We thank the reviewer for helping us improve the presentation, and we apologize for the duplicated discussion. Following the reviewer’s suggestion, we removed the repeated section and expanded the reproducibility details. Revised Appendix Tables 2, 3, 4, and 5 now report, respectively, the application architecture and graphs, randomized search space, stage-wise budgets and selection, and contamination operators. The accompanying GitHub repository will provide preprocessing code, configurations, seeds, and scripts for all three stages.

Weakness 2

Following your suggestion, the final paragraph of the revised main Introduction now states three contributions: end-to-end simplex preservation, separate Rényi orders for fitting and detection, and theory for attenuation, score separation, and graph-mediated contamination. We also added an unperturbed tonsil case study with annotation-defined minority cell populations, rather than claiming independently verified pathological anomalies. See the last paragraph of revised main Section 1 and revised main Section 4.7, especially Figures 6 and 7.

AuthorsOpenReview ↗

Response to Reviewer bNkE (part 1.5/2), question 1 -10)

Q1

In additional experiments, following your suggestion, both models use the same validation-based Rényi detection-order procedure. KL training obtains AUROC/AUPRC 0.819/0.7030.819/0.703, whereas subunit-order training obtains 0.824/0.7160.824/0.716. The modest gain supports a training-order contribution but does not establish its necessity. See revised Appendix E.1, Table 6.

Q2

We thank the reviewer for identifying this inconsistency. We audited and corrected the text: the implementation uses validation AP, reported as AUPRC, as the primary criterion, AUROC as the first tie-breaker, and the smaller order as the final tie-breaker. See the remark following Theorem 2.4 in revised main Section 2.4 and revised Appendix Table 3.

Q3

We verified that Omniglot contains 3,2463{,}246 contaminated images among 32,46032{,}460 (10.0%10.0\%), exactly two of 20 images per character; the target fraction is 10%10\% in each evaluated partition. We also expanded the qualitative interpretation of the reconstructions. See revised main Figure 4 and revised Appendix Table 2.

Q4

We corrected the ablation formula to Tg,τ(u)=(1−g)u+g softmax⁡(u/τ)T_{g,\tau}(\bm u)=(1-g)\bm u+g\,\operatorname{softmax}(\bm u/\tau); the omitted coefficient was a typographical error in the table, not in the implementation. See revised Appendix Table 8; the primary escort gate is defined in revised main Equation (4).

Q5

We thank the reviewer for raising this design issue. Following your suggestion, we clarify that the reported softmax-gate ablation uses the RD-GAE-selected τ\tau so that only the gate formula changes. This is a matched ablation, not an independent temperature search for the softmax gate; we have therefore limited its interpretation and do not claim that it is the globally best softmax-gate result. The searched RD-GAE temperature range is reported in revised Appendix Table 3, and the matched ablation shown in Table 8.

Q6

We have added the full formula to revised main Section 2, Equation (2). For the row-stochastic base operator P\bm P, the receiver consensus, sender disagreement, and normalized robust edge weight are

d_{ij}^{(\ell)}=\tfrac12\|\bm h_j^{(\ell)}-\bm c_i^{(\ell)}\|_1,\qquad \widetilde P_{ij}^{(\ell)}= \frac{P_{ij}\exp\{-\rho_gd_{ij}^{(\ell)}\}} {\sum_kP_{ik}\exp\{-\rho_gd_{ik}^{(\ell)}\}}.$$ Here $\rho_g$ controls the downweighting strength; we use $\rho_g=5$, while $\rho_g=0$ recovers $\bm P$. Self-loops enter the consensus and normalization. In the primary encoder-only model, these weights are recomputed at each encoder layer, while decoder layers use the identity graph operator. ## Q7 In updated experiments, we added a paired MNIST experiment comparing subunit-order and KL training at $0\%,5\%,10\%$, and $20\%$ training contamination, while fixing validation/test contamination at $10\%$ and scoring every model at $\lambda_{\mathrm{det}}=1.5$. As shown in revised Appendix Figure 12, subunit-order training improves AUROC at every rate and improves AUPRC at all nonzero contamination rates, with the AUPRC gain increasing to $0.022$ at $20\%$. At $0\%$, KL training has a marginally higher AUPRC. These results isolate the training order and support its benefit specifically when the training data are contaminated. ## Q8 Following your suggestion, the revised main Section 5 now identifies four limitations: the theory gives sufficient rather than universal guarantees; controlled contamination remains necessary where definitive labels are unavailable; repeated row-stochastic category mappings may attenuate profile differences, while overly strong escort sharpening may remove information; and the moderate tonsil AUROC suggests that stronger semantic representations are needed for biologically heterogeneous minority populations. It also describes MNIST and Omniglot as controlled total-mass-normalized image benchmarks rather than naturally compositional data. ## Q9 We added time and memory complexity in revised Appendix C.5. For a sparse fixed-$k$ graph, RD-GAE message passing scales linearly with the number of nodes; the main large-sample limitation of the current implementation is the $O(N^2K)$ exact graph-construction step. The revised manuscript states: > “A sparse RD-GAE forward–backward pass has time complexity $O\{S\sum_\ell(|E|d_\ell+Nd_\ell d_{\ell+1})\}$ and activation-memory complexity $O(SN\sum_\ell d_\ell)$. Thus, for a fixed-$k$ graph, $|E|=O(kN)$ and message passing is linear in $N$.” ## Q10 We clarified that each baseline uses its final Stage 3 checkpoint at the 200-epoch cap; no validation-based checkpoint is selected for the baselines. See revised Appendix Table 4 and the paragraph immediately following it.
AuthorsOpenReview ↗

Response to Reviewer bNkE (part 2/2, question 11, ethics concern)

Q11

We now state that the isometric log-ratio (ILR) transform maps a KK-part composition to K−1K-1 orthonormal log-ratio coordinates using a selected orthonormal basis (Aitchison, 1982). The ILR baselines use mean-squared reconstruction error in that coordinate space. See revised Appendix E.1, particularly the paragraph preceding Table 6.

Ethics concerns

Q1

Figure 1 is developed using LaTeX TikZ with caption text polished using AI (disclosed AI use for writing).

Q2

Thank you for the suggestion. We address this concern under Weakness 1.

Reviewer_s2GMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RD-GAE for anomaly detection on graph-linked compositional observations. Row-stochastic graph aggregation and category mappings, an escort-temperature gate, and a Dirichlet bottleneck keep the data-carrying representations on simplices. A Rényi reconstruction loss with order below one is used during fitting, while a separately selected order above one scores discrepancies after the reconstruction model is frozen. The paper derives the Dirichlet prior divergence, a conditional result for anomaly–normal score separation, and fixed-predictor contamination-propagation bounds. Experiments use synthetic perturbations of MNIST, Omniglot, microbiome, and spatial transcriptomics profiles, with comparisons against six autoencoder families and architectural ablations.

Critical strengths
  1. A coherent compositional architecture with explicitly scoped analysis (Equations (1)–(3), Equation (5), Theorems 3.1/3.3, and Appendix C). This gives the proposed architecture a reasonably transparent mathematical interpretation.
  2. Consistent empirical gains within the evaluated controlled-contamination settings. RD-GAE has the highest mean AUROC and AUPRC among the main baselines on all four datasets in Table 1. The same-order comparisons and the architectural ablations in Table 6 are useful evidence beyond a single aggregate benchmark result. The paper also appropriately distinguishes synthetic biological perturbations from independently validated natural anomalies (Appendix D).
Critical weaknesses
  1. The current evidence supports the usefulness of the combined detector, but does not establish which part of its central robust-fitting claim is necessary for the gains. In other words, the evidence does not separate the claimed robustness of subunit-order training from the benefit of a different detection score.
  2. The reported results cannot yet be tied to a fully specified, consistent evaluation protocol. Section 2.4 selects the detection order by validation AUROC, whereas Appendix E, page 17, selects by average precision and uses AUROC only to break ties. More seriously, the Table 2 note explicitly says that Omniglot's nominal contamination rate “requires confirmation from the saved payload” because the preprocessing call and injector signature differ. This is material for interpreting precision–recall results and for reproducing the experiment.
Questions for authors
  1. With the architecture, contamination, optimization/selection budget, and detection-order search held comparable, how does λtr=1\lambda_{tr}=1 with validation-selected λdet>1\lambda_{det}>1 compare against the proposed subunit-training model? Does applying the same score-order selection to the strongest simplex-output baselines preserve the reported advantage? These controls would materially affect whether I attribute the contribution to robust training, post-training scoring, or their combination.
  2. Which detection-order selection rule actually produced Table 1: validation AUROC or average precision? Please verify the realized Omniglot contamination counts from the saved experiment data and identify the graph condition and inductive/transductive protocol for each reported result. If the verified settings differ from the stated protocol, do the conclusions remain after the corresponding results are corrected?
2 comments
AuthorsOpenReview ↗

Response to Reviewer s2GM

We thank the reviewer for identifying these gaps and limitations. Following your suggestions, we conducted additional analyses and reported the results in the revised main manuscript and Appendix accordingly (see the colored text).

Weakness 1

Following your suggestion, we trained a KL RD-GAE and applied the same validation-selected superunit Rényi scoring procedure. Relative to this control, subunit training increases AUROC from 0.8190.819 to 0.8240.824 and AUPRC from 0.7030.703 to 0.7160.716. This isolates a modest contribution from the training order but does not establish that it is necessary or provide an unconditional robustness guarantee. See revised Appendix E.1, Table 6.

Weakness 2

In the updated experiments, we audited the saved outputs and corrected the description: detection order is selected by validation AP/AUPRC, with AUROC and then the smaller order as tie-breakers; the selected order is frozen before one test evaluation. Omniglot contains 3,246/32,460=10.0%3{,}246/32{,}460=10.0\% contaminated images, exactly two of 20 per character, with contamination represented in each partition. See the remark following Theorem 2.4 and Figure 4 in the revised main manuscript, and revised Appendix Tables 2 and 3.

Q1

In the revised manuscript with additional simulation, the KL-trained RD-GAE was evaluated with the same validation-selected superunit detection procedure as the subunit-trained model. The latter improves AUROC/AUPRC from 0.819/0.7030.819/0.703 to 0.824/0.7160.824/0.716, isolating a modest training-order contribution. The remaining baselines retain their canonical anomaly scores; therefore, the present analysis does not claim that each baseline is dominated under a common Rényi score. See revised Appendix E.1, Table 6.

Q2

We thank the reviewer for requesting a complete audit. Following your suggestion, we verified that Table 1 uses validation AP/AUPRC first, AUROC second, and the smaller order last; see the remark following Theorem 2.4 in the revised main manuscript and revised Appendix Table 3. Omniglot has 3,246/32,460=10.0%3{,}246/32{,}460=10.0\% contaminated images, exactly two per character; see revised main Figure 4 and revised Appendix Table 2. The reported protocols are inductive with a clean-profile graph for MNIST, inductive with an observed-profile graph for Omniglot, and transductive with metadata-based graphs for Microbiome and spatial transcriptomics; see revised Appendix Tables 2 and 4. The numerical results are unchanged, but we now restrict the biological conclusion to transductive detection and identify the MNIST clean topology as oracle.

Most of my concerns have been addressed, and I would like to raise the score.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to Reviewer Tese (Part 1/2)

We thank the reviewer for identifying these gaps and limitations. We conducted additional analyses and reported results in revised main manuscript and Appendix accordingly (see the colored text).

Weakness 1

Following your suggestion, we added an unperturbed human-tonsil case study using Slide-tags cell-type annotations. Across three nested minority-cell cohorts with prevalences 5.97%5.97\%, 10.61%10.61\%, and 17.93%17.93\%, RD-GAE attains the highest mean AUROC and AUPRC among the evaluated autoencoder baselines. These are annotation-defined minority populations, not verified technical or pathological anomalies, and we have limited the claim accordingly; see the revised main manuscript, Section 4.7 and Figure 7. The revised main manuscript states:

“The prioritized minority profiles can be mapped to their spatial neighborhoods to characterize rare cellular microenvironments and guide investigation of their functional underpinnings, although the anomaly scores alone do not establish biological function.”

Weakness 2

Following your suggestion, we repeated Stage 3 with label-free Hellinger kk-NN graphs while holding hyperparameters, folds, and seeds fixed. AUROC/AUPRC changes from 0.812/0.7130.812/0.713 to 0.812/0.7140.812/0.714 for MNIST and from 0.860/0.7060.860/0.706 to 0.857/0.6960.857/0.696 for Omniglot, despite only 3.7%3.7\% edge overlap for Omniglot. Thus, performance does not materially depend on class-restricted image graphs. We also identify MNIST and Omniglot as controlled normalized-image benchmarks rather than naturally compositional applications; see the last paragraph of the revised main Introduction and Table below. Application-specific graph conditions are reported in revised Appendix Table 2.

DataGraph constructionEdge JaccardAUROCAUPRC
MNISTClass-restricted—0.812 (0.002)0.713 (0.004)
MNISTLabel-free0.8150.812 (0.001)0.714 (0.003)
OmniglotClass-restricted—0.860 (0.005)0.706 (0.013)
OmniglotLabel-free0.0370.857 (0.004)0.696 (0.009)

Weakness 3

In the revised manuscript, we added a held-out MNIST assessment. At the selected λdet=1.5\lambda_{\mathrm{det}}=1.5, the tilted-variance condition holds for 74.5%74.5\%–81.0%81.0\% of anomalous–normal pairs across folds (mean 76.7%76.7\%), but its prevalence decreases at larger orders while AUROC first rises and then falls. We therefore retain validation selection and clarify that the theorem gives a sufficient condition, not a guaranteed operating point. For the fixed-operator no-skip analysis, the certified breakdown fraction is 6.10×10−56.10\times10^{-5}, whereas the tested perturbations reach mean clean-node displacements 0.010.01, 0.050.05, and 0.100.10 at 0.5%0.5\%–1%1\%, 2%2\%–5%5\%, and 5%5\%–10%10\% contamination, respectively. This shows that the uniform certificate is conservative; the finite stress test is not an exact worst-case breakdown calculation. See the empirical-scope remark following Theorem 3.3 in the revised main manuscript and revised Appendix C.4, Figure 8.

Weakness 4

In updated experiments, we added ILR-space IF/LOF and deep-autoencoder baselines. On the matched MNIST Stage-3 comparison, RD-GAE has the highest AUPRC (0.7160.716), while ILR-AE has the highest AUROC (0.8390.839); we therefore do not claim uniform superiority over log-ratio pipelines. Revised Appendix Table 6 reports the comparison, and revised Appendix Figure 10 reports the 10-epoch compute premium: RD-GAE uses 56.9756.97 s and 4.744.74 GB, versus 19.5419.54 s/2.222.22 GB for G-VAE and 23.8023.80 s/2.132.13 GB for G-DirVAE; see revised Appendix E.1.

Weakness 5

We evaluated Microbiome validation contamination fractions of 5%5\%, 10%10\%, 15%15\%, and 20%20\% using coarse and fine order grids while holding the fitted model, graph, preprocessing, and test set fixed. The coarse grid selects λdet=1.5\lambda_{\mathrm{det}}=1.5 in all four cases, while the fine grid selects values between 1.501.50 and 1.701.70. Selection uses validation AP/AUPRC, with AUROC and then the smaller order as tie-breakers; test labels remain excluded. This diagnostic does not eliminate selection risk but shows stability over the evaluated conditions. See the revised main manuscript, Section 2.4, and revised Appendix E.4, Figure 13.

Weakness 6

For sensitivity analysis, we evaluated the frozen MNIST model over the parameter ranges reported in revised Appendix Table 7, while holding the learned weights, preprocessing, graph, test set, and λdet=1.5\lambda_{\mathrm{det}}=1.5 fixed. Revised Appendix Figure 11 shows that performance is nearly invariant to αmin⁡\alpha_{\min} and SevalS_{\mathrm{eval}}, moderately sensitive to κ\kappa, and most sensitive to gg and τ\tau. The selected setting lies in the high-performing region, whereas weaker gating or larger temperature substantially reduces both AUROC and AUPRC; see revised Appendix E.2.

AuthorsOpenReview ↗

Response to Reviewer Tese (Part 2/2)

Q1

In the revised manuscript, we added an unperturbed human-tonsil case study based on annotation-defined minority cell populations. RD-GAE has the highest mean AUROC and AUPRC among the evaluated autoencoder baselines across all three nested target sets, while we explicitly avoid treating rarity as proof of biological abnormality. See the revised main manuscript, Section 4.7, especially the second and third paragraphs, and Figures 6 and 7.

Q2

Again in the updated manuscript, we added ILR followed by IF, LOF, AE, VAE, DirVAE, and graph-autoencoder variants under the matched Stage 3 protocol. RD-GAE has the highest AUPRC, whereas ILR–AE has the highest AUROC; we therefore no longer claim uniform superiority over log-ratio pipelines. See revised Appendix E.1, particularly Table 6.

Q3

We evaluated the tilted-variance condition on held-out MNIST pairs. At the selected order, it holds for a majority, but not all, anomalous–normal pairs, and its prevalence decreases at larger orders. This supports a conditional interpretation of the theorem and validation-based order selection rather than a universal monotonicity claim. See the empirical-scope remark following Theorem 3.3 in the revised main manuscript and revised Appendix C.4, Figure 8.

Q4

We now report matched 10-epoch wall-clock time and peak memory for RD-GAE, G-VAE, and G-DirVAE. RD-GAE uses 56.9756.97 s and 4.744.74 GB, compared with 19.5419.54 s/2.222.22 GB for G-VAE and 23.8023.80 s/2.132.13 GB for G-DirVAE. See revised Appendix E.1, Figure 10.

Q5

To evaluate, we constructed label-free Hellinger kk-NN graphs over all images and repeated Stage 3 with hyperparameters, folds, and seeds fixed. Performance does not collapse: MNIST changes from 0.812/0.7130.812/0.713 to 0.812/0.7140.812/0.714 and Omniglot from 0.860/0.7060.860/0.706 to 0.857/0.6960.857/0.696 in AUROC/AUPRC. See Table below in this response and revised Appendix Table 2 for the reported graph protocols.

DataGraph constructionEdge JaccardAUROCAUPRC
MNISTClass-restricted—0.812 (0.002)0.713 (0.004)
MNISTLabel-free0.8150.812 (0.001)0.714 (0.003)
OmniglotClass-restricted—0.860 (0.005)0.706 (0.013)
OmniglotLabel-free0.0370.857 (0.004)0.696 (0.009)

Paired Stage-3 sensitivity to label-free image graphs, with hyperparameters, folds, and seeds fixed. Edge Jaccard is $J(E_{\mathrm{class}},E_{\mathrm{LF}}) =|E_{\mathrm{class}}\cap E_{\mathrm{LF}}|/ |E_{\mathrm{class}}\cup E_{\mathrm{LF}}|$, where values near one indicate similar edge sets. Detection performance remains stable after removing class labels from graph construction.

Q6

Following your suggestion, we varied the validation-contamination fraction over 5%5\%, 10%10\%, 15%15\%, and 20%20\% and compared the prespecified coarse grid with a 0.050.05-spaced refinement from 1.51.5 to 2.252.25, holding the fitted model and test set fixed. The coarse grid selects λdet=1.5\lambda_{\mathrm{det}}=1.5 for all fractions and in all five folds; the fine grid selects only nearby values, 1.501.50–1.701.70. This supports local stability under the evaluated conditions but does not eliminate finite-sample selection risk. See revised Appendix E.4, Figure 13; the corrected AP/AUPRC-first selection rule appears in the remark following Theorem 2.4 in the revised main manuscript.

AuthorsOpenReview ↗

Follow-up Response to Reviewer Tese

Dear Reviewer Tese,

Above is our response regarding the weaknesses and questions raised in your reviewer. We appreciate your time and patience, and we have revised our manuscript according to your helpful suggestions. Please let us know if there is any additional concern that needs to be addressed. We would be happy to provide additional results and clarifications on short notice.

Regards, Submission #52331

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.