acceptodds
Under review as a conference paper at ICLR 2027

CARMA: Context-Aware Regulatory Modeling of Chromatin Accessibility

Abstract

Single-cell ATAC-seq profiles chromatin accessibility at candidate cis-regulatory elements (cCREs), but extreme sparsity and noise complicate representation learning. Foundation models learn expressive cell representations that are difficult to interpret, whereas conventional topic models yield interpretable cell mixtures and cCRE weights without learning contextual cCRE states. We introduce CARMA, which integrates contextual cCRE encoding with neural topic modeling, treating cells as documents, cCREs as words, and regulatory programs as topics. A contextual encoder computes soft cCRE-to-topic assignments, while a generative decoder reconstructs each cell's full accessibility profile using shared topic prototypes and cCRE embeddings. This design jointly learns cCRE, topic, and cell representations with an explicit cCRE-to-topic-to-cell correspondence. Across seven scATAC-seq datasets, CARMA achieved the highest cross-dataset mean cell-state separation and batch-mixing scores among the evaluated foundation models. It also outperformed the other evaluated encoders pretrained exclusively on scATAC-seq data in mean macro-F1 for cell-type prediction on every dataset. Its frozen cCRE embeddings provide competitive or superior linear-probe performance on CATlas, SCREEN, and ChromHMM annotations and capture information complementary to genomic covariates. CARMA thus makes contextual scATAC-seq representations interpretable while also improving downstream performance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
single-cell ATAC-seq, chromatin accessibility, foundation model, neural topic modeling, contextual representation learning, interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QUmZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CARMA combines contextual encoding of accessible candidate cis-regulatory elements (cCREs) with a neural topic model. A Transformer encodes sampled, TF–IDF-ranked cCRE tokens. Shared topic prototypes assign contextual tokens to topics, and assignment-weighted pooling of identity embeddings parameterizes a logistic-normal posterior over cell-level topic mixtures. A Bernoulli decoder reconstructs the full accessibility profile using the same topic prototypes and cCRE identity embeddings. This yields explicit contextual assignments, cell mixtures, and topic–cCRE log-odds coefficients.

The model is pretrained on 5.16 million human cells and evaluated on seven studies totaling 278,090 cells. Experiments assess cell-state separation, linear cell-type prediction, batch mixing, regulatory annotation recovery, motif associations, and input-hidden accessibility prediction. Matched pretraining ablations examine self-attention, parameter sharing, and posterior pooling. The results support useful representations and an inspectable predictive topic structure, with further questions about contextual biological interpretation and generalization.

Critical strengths

1. A coherent predictive connection between contextual inference and explicit topic axes, supported by matched ablations.

The same topic indices organize contextual assignments, posterior inference, and full-profile reconstruction. Consequently, the topic–cCRE coefficients have a defined interpretation in accessibility log-odds, rather than being solely post-hoc labels for an embedding. The from-scratch ablations are particularly useful: removing self-attention lowers mean AvgBIO from 0.5907 to 0.4922 and macro-F1 from 0.7580 to 0.6951; separating encoder and decoder parameters yields 0.5400 and 0.6972, respectively. Parameter sharing also reduces the trainable count by 47.0%. These controls support the proposed design beyond an architectural description.

2. Broad empirical evidence with meaningful biological and predictive controls.

Across the seven studies, CARMA's topic representation has mean macro-F1 0.7580 versus 0.7166 for EpiAgent and exceeds all three main ATAC-only pretrained encoder comparators on each study. Regulatory probes use common loci and chromosome-held-out label splits, with co-occurrence and genomic-covariate controls. Motif enrichment uses a covariate-stratified genomic background and correction over all 64×282 topic–motif-family tests. The input-hidden prediction analysis removes query loci before ranking and token selection; inferred mixtures reduce query NLL from 0.1769 to 0.1661 relative to mixtures shuffled within study, sample group, and depth strata. Together, these results establish predictive information and plausible global lineage-associated programs more convincingly than selected visualizations alone.

Critical weaknesses

1. The specifically contextual cCRE interpretation is less established than the usefulness of the cell representation and global decoder.

The distinctive object in Eq. (3) is the context-dependent assignment �0��0�. However, the annotation probes evaluate static identity embeddings �1��1�; the motif and cell-type associations examine global decoder directions �2��2� and cell mixtures; and masked reconstruction tests the predictive value of those mixtures. These are valuable results, but they do not directly show that the same cCRE's changed assignment across cellular contexts has a reproducible biological meaning. Parameter sharing establishes a common indexing and predictive structure, not by itself a validated token-level explanation.

There is also a distinction between the topic lists being assessed: Appendix I interprets cCREs ranked by �3��3�, whereas Appendix J computes coherence and diversity from �4��4�, which includes baseline accessibility. Both rankings have legitimate interpretations, but the latter does not directly validate the former. The small NPMI improvement over CARMA-Bag, −0.593 versus −0.613, is therefore limited evidence for more coherent biological signatures; sparse observations also limit this metric, as the authors acknowledge.

A same-locus, different-context analysis, with resampling stability and the existing ablation checkpoints, would address the distinguishing interpretability claim. I am not requesting a causal-regulatory theorem or wet-lab perturbation evidence: the manuscript appropriately describes candidate programs and does not claim causal identification.

2. Generalization and robustness are only partially characterized by the current evaluation protocol.

Appendix B's five seeds vary downstream evaluation while holding the pretrained checkpoint fixed. They therefore do not measure variability in pretraining or in the learned topic programs. The linear probes use cell-type-stratified folds within each study, rather than donor- or library-held-out folds. Evaluating frozen representations across seven studies is a real strength, but these probe results do not separately establish supervised transfer to new donors or experimental libraries.

The manuscript lists training-study names and evaluation-study summaries, but an accession-level mapping and explicit sample/cell-overlap audit would make the external-evaluation claim independently checkable. For cCRE annotation, the CATlas source atlas is explicitly included in pretraining, and chromosome holdout applies to probe labels rather than to the unsupervised learning of locus embeddings. This is a legitimate evaluation of annotation information in pretrained representations; it should not be interpreted as evidence of generalization to loci unseen during pretraining. I do not infer undisclosed label leakage from this overlap.

These limitations matter especially for small differences: the scETM-based model essentially matches CARMA's mean AvgBIO, 0.5906 versus 0.5907, although CARMA improves macro-F1 by 3.11 percentage points. Clearer provenance, donor/library-held-out probing, and evidence about pretraining variability would better delimit the architectural advantage.

Questions for authors

The requests below concern the current interpretation and evaluation claims. They do not require a new large-scale pretraining benchmark.

1. Contextual assignments — targeted validation or clarification. For cCREs observed in multiple cell types or states, can you show how their �0��0� assignments change, whether those changes agree with the corresponding decoder programs, and how stable they are to input-token resampling? Comparing with CARMA-Bag and the independent-decoder checkpoint would help isolate the contribution of contextualization and tying. If such evidence is unavailable, please clarify the intended interpretation of these assignments as model intermediates.

2. Program definitions — additional analysis using existing checkpoints. How do coherence, diversity, and motif associations compare when using the centered �1��1� ranking employed in the biological interpretation, including the relevant ablations? The current absolute-prediction ranking is meaningful, but evaluates a different list of regions.

3. Data separation and transfer — clarification and targeted evaluation. Please provide dataset accessions, exact training/evaluation memberships, and any sample/cell-overlap checks. On representative studies, does the macro-F1 advantage persist when probes hold out donors or libraries? For cCRE probes, please also specify whether PCA and all preprocessing are fitted on the training chromosomes and how the validation partition selects regularization.

4. Training robustness — clarification. How many independent pretraining runs support the main and ablation results? If replicated checkpoints are available, can you report variation in the main metrics and topic alignment across runs? Otherwise, please make clear that the reported seed uncertainty is conditional on one pretrained checkpoint and that topic stability across pretraining runs has not been established.

Reviewer_9rKoOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARMA combines a Transformer encoder with a neural topic model for scATAC-seq. IT leverages accessible cCREs as the representations and softly assigned to 64 shared topic prototypes. The same topic prototypes and cCRE identity embeddings parameterize a Bernoulli decoder over the full cCRE vocabulary, providing an explicit topic–cCRE geometry.

The model is pretrained on 5.16 million cells from Human-scATAC-Corpus and then evaluated on seven case studies. The paper also evaluates cCRE annotation prediction, biological motif associations, topic coherence/diversity, masked reconstruction, architectural ablations, and inference efficiency.

Critical strengths
  • The empirical study is broad. It covers seven datasets, three cell-level criteria, four cCRE annotation tasks, biological interpretation, masked targets, efficiency, and several ablations.
  • The masked-target experiment is valuable. Removing query cCREs before input selection and comparing inferred, shuffled, and uniform mixtures provides credible evidence that cell mixtures retain predictive information.
  • The chromosome-held-out test and covariate-augmented controls are more considerable than ordinary random cCRE splits.
Critical weaknesses
  1. CARMA appears to be an incremental composition of established components: a Transformer cCRE encoder, logistic-normal neural topic inference, prototype-based token assignment, and an embedding-factorized decoder. The main architectural distinction is sharing the topic prototypes and cCRE identity embeddings between contextual assignment and reconstruction. While sensible, this is primarily a weight-tying modification rather than a fundamentally new modeling principle. Contextualized topic models already produce token-level topic assignments, while ETM-derived methods—including the scATAC-specific GFETM—already provide embedding-factorized, interpretable topic decoders. The paper does not directly demonstrate that CARMA’s particular coupling enables capabilities unavailable to these prior approaches. In particular, it does not analyze whether the same cCRE receives meaningfully different assignments in different cellular contexts. Without such evidence or comparisons against GFETM/cisTopic, the work reads primarily as a large-scale transfer and integration of existing ideas to scATAC-seq.
  2. The novel object is the context-dependent token assignment rbℓkr_{b\ell k}. However, the biological analyses primarily interpret the global decoder coefficient DkjD_{kj}, cell-level mixture θb\theta_b, and static identity embedding CjC_j. The paper never shows that the same cCRE receives meaningfully different topic assignments in different cellular contexts, whether these changes correspond to known biology, or whether the assignments are stable under repeated input subsampling.
  3. The matched scETM baseline reaches essentially identical mean AvgBIO—0.5906 versus 0.5907—and substantially higher topic diversity, although CARMA improves macro-F1 and batch mixing. This makes the advantage over conventional topic modeling more nuanced than the main presentation suggests. More importantly, the paper cites but does not experimentally compare with GFETM or cisTopic. GFETM is a particularly relevant, peer-reviewed scATAC-seq embedded topic model with interpretable cell-topic and region-topic structure; cisTopic is the established classical baseline for cis-regulatory topic modeling. Their omission weakens both the novelty and empirical positioning. GFETM and cisTopic are not merely peripheral alternatives.
  4. The authors should provide more detals about the seven datasets. Exact accessions, preprocessing, donor composition, and an explicit audit against all 33 pretraining studies are missing.
  5. Leiden resolution is chosen using the known number of annotated cell types. The representations also have different dimensions and preprocessing: ChromFound alone receives PCA to 50 dimensions, while the other models use their native readouts. It is unclear whether features were standardized before Euclidean kNN construction. Results should be checked under matched dimensions, standardized features, multiple neighborhood sizes, and an evaluation that does not use the true class count.
  6. All principal uncertainty estimates hold the pretrained checkpoint fixed and vary downstream folds or clustering seeds. These very small standard deviations therefore do not measure model-training stability. The paper’s claims of outperforming competing architectures would be more convincing with multiple CARMA pretraining seeds, at least for the main model and decisive ablations, or with uncertainty over datasets.
Questions for authors
  • Can you provide an explicit accession-level overlap table between every pretraining and evaluation dataset?
  • Can you directly evaluate contextual assignments rbℓkr_{b\ell k}? For example, does the same cCRE shift topics across cell types in a reproducible and biologically meaningful manner?
  • How stable are assignments and cell mixtures across repeated random 8,192-cCRE input samples?
  • Why are GFETM and cisTopic absent from the empirical topic-model comparison?
  • What are the results under donor-, sample-, or batch-held-out cell-type probes?
  • How much variation arises from independent CARMA pretraining runs?
  • Does the weight-sharing advantage persist against a parameter-matched independent decoder?
  • How is exact full-vocabulary reconstruction implemented efficiently, and what were total training time and compute?
  • Are all representations standardized before kNN construction, and do conclusions persist with matched dimensionality and alternative neighborhood sizes?
Reviewer_5vavOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The Authors introduce CARMA mechanism integrating contextual cCRE encoding with neural topic modeling, treating cells as documents, cCREs as words, and regulatory programs as topics. A contextual encoder computes soft cCRE-to-topic assignments, while a generative decoder reconstructs each cell’s full accessibility profile using shared topic prototypes and cCRE embeddings. This design jointly learns cCRE, topic, and cell representations with an explicit cCRE-to-topic-to-cell correspondence. The Authors claim that across seven scATAC-seq datasets, CARMA achieved the highest cross-dataset mean cell-state separation and batch-mixing scores among the evaluated foundation models. The also claim that it outperformed the other evaluated encoders pretrained exclusively on scATAC-seq data in mean macro-F1 for cell-type prediction on every dataset.

Critical strengths

The paper focuses on the problem of critical importance to the molecular design: learning contextual cell representations. It addresses some of the main limitations on the approaches based on the foundation models: learning expressive cell representations that are difficult to interpret. The empirical verification is very thorough: stretching from pre-training (small computational footprint and a compact model of < 400M parameters are additional strengths), through identifying regulatory information to experiments involving biological interpretation of topic axes. Gains obtained over competitors regarding linear annotation recovery from cCRE representations (e.g. ~0.2 over second best model on CATlas AP) are impressive.

Critical weaknesses

The paper would benefit from the more thorough description of the Transformer model applied (Sec. 4.1), e.g. type of the positional encodings used (e.g.: are RPEs applied ?) as long as ablations over different architectural choices of the encoder on the presented downstream results. The analysis of the computational efficiency of CARMA over other approaches is missing and would be a valuable addition to the manuscript.

Questions for authors

The Reviewer would like to ask the Authors the address questions mentioned in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.