CARMA: Context-Aware Regulatory Modeling of Chromatin Accessibility

Single-cell ATAC-seq profiles chromatin accessibility at candidate cis-regulatory elements (cCREs), but extreme sparsity and noise complicate representation learning. Foundation models learn expressive cell representations that are difficult to interpret, whereas conventional topic models yield interpretable cell mixtures and cCRE weights without learning contextual cCRE states. We introduce CARMA, which integrates contextual cCRE encoding with neural topic modeling, treating cells as documents, cCREs as words, and regulatory programs as topics. A contextual encoder computes soft cCRE-to-topic assignments, while a generative decoder reconstructs each cell's full accessibility profile using shared topic prototypes and cCRE embeddings. This design jointly learns cCRE, topic, and cell representations with an explicit cCRE-to-topic-to-cell correspondence. Across seven scATAC-seq datasets, CARMA achieved the highest cross-dataset mean cell-state separation and batch-mixing scores among the evaluated foundation models. It also outperformed the other evaluated encoders pretrained exclusively on scATAC-seq data in mean macro-F1 for cell-type prediction on every dataset. Its frozen cCRE embeddings provide competitive or superior linear-probe performance on CATlas, SCREEN, and ChromHMM annotations and capture information complementary to genomic covariates. CARMA thus makes contextual scATAC-seq representations interpretable while also improving downstream performance.