acceptodds
Under review as a conference paper at ICLR 2027

JEDI: Jointly Embedded Inference of Neural Dynamics

17 views

Abstract

Animal brains flexibly and efficiently achieve many behaviors with a single neural network. A core goal in modern neuroscience is to map the mechanisms of the brain’s flexibility onto the dynamics underlying neural populations. However, identifying task-specific dynamical rules from limited, noisy, and high-dimensional neural recordings remains a major challenge, as experimental data often provide only partial access to brain states and dynamical mechanisms. While recurrent neural networks (RNNs) directly constrained by neural data have been effective in inferring underlying dynamical mechanisms, they are typically limited to single-task domains and struggle to generalize across behavioral conditions. Here, we introduce JEDI, a hierarchical model that captures neural dynamics across tasks and contexts by learning a shared embedding space over RNN weights. This model recapitulates individual samples of neural dynamics while sharing statistical strength across trials and conditions, uncovering shared structure across conditions in a single, unified model. Using simulated RNN datasets, we demonstrate JEDI accurately learns robust, generalizable, condition-specific embeddings, and recovers the fixed point structure of a task-trained network. Finally, we apply JEDI to motor cortex recordings during monkey reaching to extract mechanistic insight into the neural dynamics of motor control. Our work shows that joint learning of contextual embeddings and recurrent weights provides scalable and generalizable inference of brain dynamics from neural recordings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural Dynamics, Recurrent Neural Networks, Hypernetworks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_K3vTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced JEDI, a hierarchical hypernetwork model that generates low-rank autonomous RNNs to fit neural data, and the model can also learn a context embedding for each trial. The model is evaluated on both simulated data and real data and the model shows more interpretability.

Critical strengths
  1. The approach is clear. The hypernetwork allows trial-specific dynamics while sharing parameters across trials. Generating low-rank connectivity provides a practical structural constraint and makes the learned networks good for dynamical analysis.
  2. The authors did evaluation beyond reconstruction accuracy. The paper investigates embedding structure, perturbation robustness, eigenvalue spectra, fixed points, and Lyapunov exponents.
Critical weaknesses
  1. The experiments do not establish that varying recurrent weights is necessary to explain contextual differences. In the simulated data, the teacher’s connectivity is fixed across tasks. However, JEDI absorbs these differences into context-dependent connectivity. I appreciate the authors for acknowledging this limitation, but it directly affects their central interpretation. A shared-weight RNN with context-dependent inputs would be an important baseline.
  2. The fairness of the model comparisons requires further clarification. Although the methods use the same embedding dimensionality, their parameter counts may differ substantially. Please report the total number of trainable parameters. Also, section E.4 says baselines get the same first bin as JEDI, could the authors discuss how you give this information to VAE and LFADS?
  3. In Fig. 5D, exact values and tested pairs are not reported, and CEBRA and LFADS appear to classify reach direction better than JEDI. Could the authors discuss more?
  4. The claim that JEDI learns a “clear ring-like structure” appears to rely on 2D PCA visualization. However, their circular geometry is less evident. Could the authors quantify it?

Minor:

  1. A.2: line 795 has W_ext and line 800 has another W_ext, are they different?

Overall, I appreciate this paper’s idea of using a hypernetwork over learned context embeddings to generate condition-specific, low-rank RNN. However, several weaknesses should be addressed to strengthen the paper.

Questions for authors

Please see section Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer K3vT (1/2)

Thank you for your careful review. We appreciate your note that the approach is clear and that the evaluation goes beyond reconstruction accuracy. We address each of your points below. Unless noted, R² is on held-out trials, with ranges over ≥3 seeds.

W1. The experiments do not establish that varying ...

We agree that this is an important baseline, and we would kindly refer you to CQ1, which reports it at matched capacity, together with teachers whose ground truth is known. Where connectivity differs across conditions only JEDI generalizes (0.95 vs −0.52), and where only inputs differ, JEDI given the input keeps its connectivity shared.

W2. The fairness of the model comparisons requires ...

The number of trainable parameters are listed in the table below, where synthetic data have 200 neurons and monkey data 117. The context-input RNN is matched to JEDI, and JEDI has 14–42× fewer parameters than the VAE. We also trained parameter-matched versions of the VAE and LFADS (below). We matched parameters by changing only layer widths, keeping each architecture otherwise unchanged. The context-input RNN shares one low-rank connectivity across conditions, and the width of the hidden layer that maps each context to its constant input is set so that its total parameter count equals JEDI's. For the VAE, the hidden layers of the encoder and decoder shrink from 128 to 9 units on the monkey data and to 3 on the synthetic data. For LFADS, the initial-condition and controller-input encoders, the controller, the generator, the initial-condition dimension and the factors grow from 16 to 60 units on the monkey data and to 57 on the synthetic data.

ModelSyntheticMonkey
Context-input RNN260,213213,475
VAE10.9M3.0M
LFADS51,32033,973
Meta-SSM, one latent per condition675,104578,160
Meta-SSM, one latent per trial469,808457,690
LtrRNN590,195573,035
VAE, parameter-matched295,582222,782
LFADS, parameter-matched259,395214,153
JEDI (Ours)260,176213,478

On the first bin, thank you for kindly checking this. App. E.4 was inaccurate. In the submission, only JEDI started each held-out trial from its recorded first bin. The VAE decoder received only its latent, and LFADS inferred its initial condition from the whole trial during fitting and used the condition-mean initial condition on held-out trials. We will correct E.4. For the rebuttal, we retrained both baselines so that they also receive each trial's first time bin. The VAE decoder takes [z; x(0)], and the LFADS initial-condition encoder sees only the first bin instead of the whole trial, with its controller set to the direction mean. All models are then matched on the information available at t = 0, and JEDI leads (0.39–0.40 vs 0.37–0.38 for the VAE and 0.35 for LFADS). When every model instead sees the whole test trial, LFADS benefits most, since its controller adds trial-specific input at every time step, yet JEDI still leads (0.53 vs 0.49). The table below reports held-out R² on monkey execution in both settings. Its first column uses baselines retrained to receive the first bin, and the second column uses the original baselines. The parameter-matched rows use a VAE hidden layer of 9 units and LFADS sizes of 60.

MethodFirst bin + direction context or latentContext or latent fitted to test trial
VAE0.37–0.380.41–0.42
LFADS0.350.49
VAE, parameter-matched0.31–0.320.01–0.33
LFADS, parameter-matched0.370.52–0.53
JEDI (Ours)0.39–0.400.53

With parameters matched to JEDI, LFADS improves and reaches JEDI on the whole test trial (0.52–0.53 vs 0.53), while JEDI still leads with the first bin (0.39–0.40 vs 0.37); the smaller VAE becomes worse and unstable.

Action. App. B.7 of the revised paper lists the parameter counts, and App. E.4 is corrected to state how each baseline receives the first bin, with this table.

AuthorsOpenReview ↗

Response to Reviewer K3vT (2/2)

W3. In Fig. 5D, exact values and tested ...

We recomputed Fig. 5D on all 208 trials with 10× repeated 5-fold cross-validation. The table gives the exact values (reach-direction accuracy, mean ± s.d. over the cross-validation splits) and the test of each method against JEDI (corrected resampled t-test [1], Holm-corrected).

MethodAccuracyvs JEDI
JEDI (ours)0.914 ± 0.029–
LFADS, initial condition0.918 ± 0.035p = 1
LFADS, initial condition + controller0.946 ± 0.023p = 0.39
CEBRA (trained with the direction labels)1.000 ± 0.000p = 7×10⁻⁶

LFADS is not significantly better than JEDI. CEBRA is better because it is trained with the direction labels to separate them [2], whereas JEDI's context is learned without labels, only to reconstruct each trial's activity.

Action. App. F.7 of the revised paper reports the exact Fig. 5D values and the tested pairs, and Section 3.4 states that direction is near ceiling for every representation.

W4. The claim that JEDI learns a “clear ...

We agree, and we are happy to have quantified it. On held-out contexts, we measured whether the direction centroids lie in cyclic order, RSA with angular distance [3], and whether the centroids form a loop. The loop is measured with persistent homology, as the lifetime of the centroids' largest one-dimensional hole (H1), tested against a label-shuffle null (500 shuffles; minimum p = 0.002), as for head-direction rings [4].

ModelCyclic order↑RSA↑Loop (H1) p↓
JEDI8/80.68–0.810.002

The direction centroids lie in a ring in every seed. All eight are in cyclic order, distances follow angle, and the loop is significant. The loop also stays significant on native time bins and in a second monkey (all 11 fits, p ≤ 0.018; RSA 0.60–0.83).

Action. The ring table is in App. F.8 of the revised paper, and Section 3.4 cites these metrics for the ring claim.

Minor 1. A.2: line 795 has W_ext and line ...

Minor. W_ext. Thank you for kindly spotting this. Both denote the same input matrix; we will unify the notation and dimensions.

Action. App. A.2 of the revised paper uses the single symbol W_ext with one stated dimension.

Thank you again for your precise questions. If our responses address your concerns, we would be grateful if you would consider raising your score.

References

[1] Nadeau, Claude, and Yoshua Bengio. “Inference for the Generalization Error.” Machine Learning, vol. 52, 2003, pp. 239–281.

[2] Schneider, Steffen, et al. “Learnable Latent Embeddings for Joint Behavioural and Neural Analysis.” Nature, vol. 617, 2023, pp. 360–368.

[3] Kriegeskorte, Nikolaus, et al. “Representational Similarity Analysis: Connecting the Branches of Systems Neuroscience.” Frontiers in Systems Neuroscience, vol. 2, 2008, p. 4.

[4] Chaudhuri, Rishidev, et al. “The Intrinsic Attractor Manifold and Population Dynamics of a Canonical Cognitive Circuit across Waking and Sleep.” Nature Neuroscience, vol. 22, 2019, pp. 1512–1520.

I thank the author for answering my questions and addressing my concerns. The quantification results, especially for parameter matched comparison is highly helpful, although it may show JEDI is only performing a little better. I now have no further questions to the author and will consider raising my score to positive after discussion with other reviewers and the AC in the next phase.

Reviewer_wr71OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes JEDI, which jointly learns trial-specific context embeddings and a shared hypernetwork that generates low-rank, neuron-indexed RNN connectivity. These autonomous RNNs reconstruct neural trajectories from observed initial states. Experiments on synthetic networks, a task-trained MemoryPro network, and macaque motor-cortex recordings evaluate embedding structure, held-out reconstruction, and inferred dynamical properties.

Critical strengths
  1. The shared hypernetwork provides a simple way to model context-dependent dynamics while sharing information across trials. Generating low-rank connectivity directly in neuron-indexed space also enables spectral, fixed-point and Lyapunov analyses without inverting a latent-to-neuron readout.

  2. The validation covers synthetic benchmarks, a frequency sweep, quantitative fixed- point comparisons and motor-cortex recordings. The supplementary rank, capacity and regularization analyses are informative, and unfavorable results, including poor context-phase recovery, are reported openly.

  3. The embeddings capture meaningful structure without behavioral-label supervision. They organize by reach direction in motor cortex and distinguish generating ranks within synthetic tasks, providing evidence that the shared representation captures systematic differences between conditions.

Critical weaknesses
  1. The autonomous formulation cannot distinguish external drive from intrinsic dynamics. The synthetic teacher has fixed connectivity across conditions, yet JEDI represents input-induced differences through trial-specific weights. Without an input- aware comparison, the learned spectra and motor-cortex stability changes remain properties of an effective model rather than validated mechanisms of the original system.

  2. Fixed-point geometry does not establish recovery of attractor dynamics. Pooling fixed points from trial-specific networks can produce a ring without any single network supporting a ring attractor. Similarly, trajectory endpoints and near-zero Lyapunov estimates require additional stability and perturbation analyses to support the proposed mechanisms.

  3. The comparisons do not isolate the benefit of weight modulation over input modulation. Moreover, Meta-SSM and LtrRNN are evaluated only on fixed-point geometry, while LFADS is assessed through its initial-condition embedding despite its controller carrying task information. These choices limit conclusions about JEDI’s comparative advantage.

  4. Mechanistic validation relies on fully observed synthetic systems, whereas real recordings provide only partial observations. No neuron-subsampling experiment tests this mismatch. The motor-data splits, animal/session breakdown and preprocessing are also insufficiently specified, and uncertainty across fits is not established for the main mechanistic findings.

Questions for authors
  1. Please compare JEDI with a shared low-rank RNN in which the same context embedding modulates a constant input or bias, under matched capacity and training. Would adding explicit time-varying inputs change the predictive advantage or the inferred dynamical properties?
  2. Within a single inferred MemoryPro memory-phase network, do different initial states preserve the full ring, including its local stability and perturbation responses? For the motor-cortex criticality claim, please also report the Lyapunov estimation horizon and convergence checks. 3.How do reconstruction and mechanistic recovery change when progressively smaller subsets of the teacher’s neurons are observed? Please assess fixed-point geometry and local stability, since good reconstruction alone would not resolve the partial-observation concern.
  3. Please evaluate Meta-SSM and LtrRNN on held-out prediction and embedding quality under matched protocols. For LFADS, how do the conclusions change when the evaluated representation includes controller information rather than only the initial condition embedding?
  4. Please specify the trial splits, animal/session breakdown, preprocessing and phase-specific context construction, and identify which metrics use held-out trials. Please also clarify the MemoryPro teacher architecture and confirm that JEDI and all baselines are fitted to identical trajectories.
8 comments
AuthorsOpenReview ↗

Response to Reviewer wr71 (1/3)

Thank you for your detailed review. We appreciate your recognition that neuron-indexed connectivity makes spectral, fixed-point and Lyapunov analyses direct, and that the embeddings capture task structure without labels. We would like to address your concerns about mechanism, and we respond to each point below.

W1. The autonomous formulation cannot distinguish external drive ...

We would kindly refer you to CQ1, where we test what JEDI's condition-specific weights capture against ground truth, with and without explicit inputs, and against the requested shared-weight baseline. When the connectivity truly differs across conditions (T-w, T-mix), JEDI recovers the teacher's eigenvalues and frequencies. When only the input differs (T-in) and that input is given to JEDI, it keeps its connectivity shared, as the teacher does, and without the input its spectra form an internal model of the input. We therefore present J_c as effective couplings, and the spectral and stability results as properties of the fitted dynamics.

Q1. Please compare JEDI with a shared low-rank ...

We would kindly refer you to CQ1, which reports the requested matched baseline and the effect of explicit time-varying inputs on both the predictive advantage and the inferred spectra. In short, at matched parameters (260,213 vs 260,176) JEDI predicts held-out trials better wherever the connectivity differs across conditions (T-w 0.95 vs −0.52, T-mix 0.94 vs −0.30, MemoryPro 0.80 vs 0.49), and the two models tie when only a known input differs (T-in 0.994 vs 0.996). With the task inputs given to both, JEDI keeps its advantage on MemoryPro (0.85 vs 0.58). The inputs change the inferred spectra only where the input is the true cause. On T-in the eigenpair at the input frequency disappears, while on the monkey data JEDI still uses direction-specific weights and both phases stay stable.

W2. Fixed-point geometry does not establish recovery of ...
Q2. Within a single inferred MemoryPro memory-phase network, ...

We would kindly refer you to CQ2 for MemoryPro, where every fixed point is tested for stability and perturbation response as a function of the cue angle. Here we add the same tests for the monkey data and give the Lyapunov details.

Monkey fixed points. For each trial's network, we ran the trial forward, refined its end state into a fixed point, and computed the eigenvalues of the Jacobian there. A fixed point is found in 81–94% of preparation trials and 94–100% of execution trials, and 96–100% of them are stable. The execution fixed points have slower directions than those in preparation (median slowest eigenvalue −0.06 to −0.09/τ vs −0.13 to −0.24/τ), so small perturbations along them decay over 1–2 s rather than 0.4–0.8 s, which matches the Lyapunov result below. Continued past the end of the trial, the model trajectories settle at these fixed points, so we will describe them as the states the trajectories head towards.

Lyapunov horizon and convergence. Following your suggestion, we now compute λ1 by QR (Benettin) iteration from the recorded first state of 64 trials per fit, over horizons from 0.2 to 50 s. The Lyapunov exponents are themselves a perturbation analysis, since they measure how small perturbations grow or decay along each trajectory. The table below gives λ1 on the time-normalized data, as in the paper, as a range over 7 execution and 6 preparation fits.

HorizonExecution λ1 (/s)Preparation λ1 (/s)
5 s−0.97 to −1.30−1.16 to −1.35
10 s−0.69 to −1.14−1.12 to −1.28
20 s−0.50 to −1.05−1.16 to −1.29
50 s−0.38 to −0.94−1.13 to −1.26

Preparation has converged by 5 s. Execution is still rising slowly at 50 s, so its limit lies even closer to zero. Below 5 s the estimate is dominated by the decay of the initial transient and changes quickly with the horizon, so we report the long-horizon values. From 20 s on, the execution and preparation ranges no longer overlap. On native time bins the ordering persists (−0.73 to −0.77 vs −0.79 to −1.12/s at 50 s). On the paper's fits, the gain at which λ1 reaches zero is 1.09–1.16 for execution vs 1.19–1.48 for preparation. Both phases are therefore stable, with slow modes of 1–3 s. These are near-marginal dynamics on the stable side of the edge of chaos.

Action. App. D of the revised paper reports the horizon, the convergence table and the gain sweep, and App. F.4 gives the stability of the monkey fixed points. Section 3.4 describes the fixed points as the states the trajectories head towards, and restricts the closer-to-the-boundary claim to long horizons, with the native-bin results next to it.

AuthorsOpenReview ↗

Response to Reviewer wr71 (2/3)

W3. The comparisons do not isolate the benefit ...
Q3 (Meta-SSM, LtrRNN, LFADS). Please evaluate Meta-SSM and LtrRNN on held-out ...

Thank you for these suggestions. We address the three points in turn.

  • Weight vs input modulation. CQ1 isolates this with a shared low-rank RNN whose context sets a constant input, matched to JEDI's parameter count (260,213 vs 260,176). The two models differ only in where the context acts, on the weights or on the input. JEDI predicts held-out trials better wherever the connectivity truly differs across conditions (T-w 0.95 vs −0.52, T-mix 0.94 vs −0.30) and on MemoryPro (0.80 vs 0.49). On monkey execution it also exceeds the context-input RNN on per-trial held-out R² (+0.053, p < 10⁻¹¹; see K3vT W3). When only a known input differs, the two tie (T-in 0.994 vs 0.996), as expected.
  • Meta-SSM and LtrRNN under matched protocols. All models are fitted to the same training trials and scored on the same held-out trials, with either the condition's context (JEDI) or latent (other models), or one fitted to the test trial. In the table below, each cell gives the condition's context or latent / the one fitted to the test trial. For LFADS with its controller, the first number uses the condition's mean controller input. JEDI, Meta-SSM and LtrRNN start each trial from its first time bin. LFADS is used unmodified, as in its original implementation, so it does not take the first time bin. Its encoder infers the initial condition from the whole test trial, and in the condition-only setting we give it the direction-mean initial condition. With the condition's context or latent, JEDI predicts best on all three datasets. When the representation is fitted to the test trial, JEDI also leads on the monkey data and MemoryPro, while LFADS with its controller reaches 1.00 on T-w, because its controller reads the test trial at every time step.
MethodMonkey executionMemoryProT-w
LtrRNN0.30 / 0.340.76 / 0.77−0.04 / 0.06
Meta-SSM, one latent per condition0.23 / 0.230.79 / 0.790.64 / 0.64
Meta-SSM, one latent per trial0.17 / 0.180.75 / 0.760.82 / 0.80
LFADS, initial condition only0.33–0.34 / 0.40–0.410.78 / 0.79−0.58 to −0.17 / 0.24–0.43
LFADS, with controller0.34–0.35 / 0.48–0.490.77–0.78 / 0.81–0.82−0.14 to −0.12 / 1.00
JEDI (Ours)0.39 / 0.530.80 / 0.850.95 / 0.83
  • Embedding quality (monkey execution, 48 held-out trials). We asked how well each model's per-trial representation captures reach direction (table below). Accuracy is that of a logistic-regression decoder of direction, trained on the training trials. Cyclic order counts how many of the 8 direction centroids have their neighbouring directions as neighbours in the first two principal components (8/8 is a perfect ring). RSA is the Spearman correlation between the distances of the centroids and their angular distances. Meta-SSM is shown with one latent per trial, since its per-condition version reads the condition label. JEDI and LFADS with its controller both order all eight directions in a ring, with similar accuracy and RSA, while Meta-SSM's latents do not form a ring.
MethodAccuracy↑Cyclic order↑RSA↑
Meta-SSM, one latent per trial0.792.7/80.22
LFADS, with controller0.928/80.71–0.84
JEDI (Ours)0.88–0.968/80.68–0.81

Including the controller improves LFADS (prediction 0.49 vs 0.41; decoding 0.92 vs 0.84 for the initial condition alone), confirming that it carries task information. JEDI still predicts held-out trials best (0.53), its decoding accuracy is not significantly different (10×5-fold CV, 0.914 vs 0.946, p = 0.39), and JEDI's contexts and LFADS's latents both order all 8 directions around a ring. We are therefore glad to see that our conclusions hold with the controller included.

Action. The baseline table and these embedding metrics are in App. F.5 of the revised paper, which reports LFADS both with and without its controller.

AuthorsOpenReview ↗

Response to Reviewer wr71 (3/3)

W4 (partial observation). Mechanistic validation relies on fully observed synthetic ...
Q3 (neuron subsets). How do reconstruction and mechanistic recovery change ...

We fitted JEDI while observing only a fraction of the teacher's neurons, and compared each fit with the fit to all neurons on three things. These are how well it predicts held-out trials, how close its eigenvalues are to the teacher's, and whether its fixed points sit at the right place on the ring and are stable (table below, means over 3 seeds; 200 neurons in T-w and 150 in MemoryPro). Down to 12.5–25% of the neurons, prediction, eigenvalues and stability are essentially unchanged; only the placement of the fixed points drops slightly (0.99 with all neurons, 0.92 at 25% and 0.90 at 12.5%). The slowest eigenvalue stays near −0.6/τ down to 25% and moves only slightly toward zero at 12.5% (−0.45/τ), so every fixed point stays clearly stable and no spurious slow modes appear.

ExperimentMeasure100%50%25%12.5%
Teacher whose connectivity differs across conditions (T-w)Held-out R²0.950.970.950.95
Distance of JEDI's eigenvalues from the teacher's0.130.150.180.13
Frequency teacher with fixed connectivity and sine inputs (T-in, 200 neurons)Held-out R² (forecast)0.720.760.840.82
Slope of eigen-frequency vs input frequency (ideal 1)1.150.980.981.00
MemoryProHeld-out R²0.800.830.770.77
Fixed points at the right angle on the ring (circular correlation with the cue)0.990.980.920.90
Slowest eigenvalue at the fixed points (/τ)−0.57−0.59−0.60−0.45

Action. The subsampling results are in App. F.6 of the revised paper.

W4 (data and uncertainty). The motor-data splits, animal/session breakdown and preprocessing ...
Q4. Please specify the trial splits, animal/session breakdown, ...

W4 / Q4. Data and uncertainty. Thank you for this request. We are happy to provide the evaluation protocol and the datasets in the two tables below.

QuestionAnswer
Uncertaintyranges over ≥3 seeds (monkey data also 5-fold CV)
Testsdecoding: corrected resampled t-test; per-trial R²: paired Wilcoxon; both Holm-corrected. Fixed points and ring metrics: permutation tests
Held-out trialsall R² and decoding results; dynamical analyses use the fitted networks
Same data for all modelsyes, same trials and splits
Contextsone per trial and task phase
DataUnitsTrain / testPreprocessing
Monkey M117 (51 M1, 66 PMd)160 (first 20 per direction) / 4810 ms smoothed rates; preparation 20 bins; execution time-normalized to 150 samples
Monkey C255 (63 M1, 192 PMd)160 / 49native time bins
MemoryPro150, read out from a 128-unit GRU teacher720 / 3594 phases of 75 steps

Action. App. A.1 and A.3 of the revised paper give the splits, sessions, preprocessing and the MemoryPro teacher as above, and App. F.9 gives the evaluation protocol.

Thank you again for the thorough review. If these new analyses address your concerns, we would be grateful if you would reconsider your assessment.

Reviewer_2KrdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the authors introduce JEDI, a hierarchical model of neural dynamics aimed at capturing variability across different tasks and contexts. they accomplish this by learning a shared embedding space over perturbations to the RNN weights. the authors demonstrate the efficacy of their approach both on simulated data and neural data from a monkey while it performed a reaching task.

Critical strengths

the work is particularily relevant, especially as neural datasets are scaling up and incorporating more naturalistic behavior, making it hard to fit models that capture trial, task, or contextual variability — which the authors have shown they are able to do to a certain extent with their model.

additionally, the authors had a good variety of experiments. the authors demonstrate on real monkey reaching data that their model uncovers task related structure previously uncovered in the literature.

the writing is clear, the progression of the experiments is well thought out, and relevant literature seems cited.

Critical weaknesses

in the toy example for fig2, it would have been interesting to see how JEDI for other ranks r performs. for the toy example of fig4, it would have been good to also compare to the structure uncovered by another method (for example the VAE which had high R2 in the other example). the toy examples also have fairly simple structures, i am wondering if there is a mixture of different topological structures (perhaps fixed point structures in one subspace and oscillatory structures in another) if the method would be able to recover it. in my opinion, demonstrating the efficacy of this method on a harder toy example where the ground truth is known would help to alleviate some concerns.

Questions for authors

im wondering if the authors can clear up the meaning of the sentence “While VAE embeddings collapse at high noise level” — i am having trouble interpreting this because from the Fig.3 plot it looks like VAE is the strongest at high noise level.

In Fig 4F it does not look like the actual structure was well recovered. this also makes me wonder how adept the method is at handling more complicated dynamical systems structures, have the authors considered harder toy examples?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2Krd

Thank you for your positive and constructive review. We appreciate your note that modelling trial, task and context variability matters as datasets grow, and that the experiments and writing were clear. We respond to each of your points below.

W1 (ranks). in the toy example for fig2, it ...

We would kindly like to point to the supplementary material, which reports this analysis. App. E.1 (Fig. S1) sweeps JEDI's rank r from 2 to 200 on the Fig. 2 data, and every rank fits well at the chosen hypernetwork sizes. App. E.3 varies the teacher's rank from 1 to 100. The effective rank of the generated J tracks it (correlation 0.95–0.97), so r acts as an upper bound rather than a parameter to tune.

Action. Section 3.1 of the revised paper points to App. E.1 and E.3 for this sweep and states that r acts as an upper bound on the rank.

W2 (another method). for the toy example of fig4, it ...

We agree that such a comparison would be informative. A VAE, however, decodes its latent with a feedforward network, so it does not define a dynamical system whose spectrum or fixed points could be compared with Fig. 4. We would kindly point to Table 1 of the paper, which compares the MemoryPro fixed-point geometry of Fig. 4 with two dynamical baselines, Meta-SSM and LtrRNN. JEDI has the lowest disparity in every phase with ring structure. Action: Section 3.3 of the revised paper states that the VAE defines no dynamical system to compare, and points to Table 1 for the dynamical baselines.

W3 (harder toy example). the toy examples also have fairly simple ...

We built such a teacher (T-mix). In each condition, a limit cycle in one subspace coexists with one or two fixed-point attractors in another. JEDI recovers both (table below, with held-out R² and the median frequency error over a 30 s free run). It recovers the frequency within 10% accuracy in every network, and the number of attractors in every condition. The context-input RNN merges the two attractors into one in all 12 bistable-condition networks, and the single RNN adds 3 spurious attractors.

MethodR²↑Freq.\ error↓
Single RNN0.0944%
Context-input RNN−0.3046%
JEDI (Ours)0.941.1%

Action. The T-mix teacher and this table are in App. F.2 of the revised paper, which reports the recovered frequencies and attractor counts against both baselines.

Q1. im wondering if the authors can clear ...

We agree that the sentence was unclear. By “collapse” we meant that the VAE's reconstructions converge to the mean trajectory across tasks, not that its R² drops and stays constant. The VAE is trained with a KL penalty that pulls every posterior toward the N(0,I) prior, and it decodes noisy samples z = μ + σε, so its decoder learns to map latents that carry little task information to the average trajectory. Large perturbations move the embeddings into that region, and the decoder then returns a near-mean trajectory for every trial. That output still shares the time course common to all tasks, which kept the VAE's R² positive in Fig. 3, although the reconstruction no longer depends on the task. LFADS does not collapse in this way, because the perturbation does not reach all of its trial information. We perturb its initial condition, which sets the initial state of the generator GRU, but its controller GRU still receives the encoder's unperturbed inputs at every time step and passes inferred inputs to the generator. The trial's identity therefore still reaches the generator throughout the trial, so its reconstructions stay tied to the trial instead of converging to the mean. JEDI fails differently because a perturbed context generates a different recurrent network, whose error compounds along the trial. Action. Section 3.1 of the revised paper replaces the sentence with this explanation of how each model fails under perturbation.

Q2. In Fig 4F it does not look ...

We would kindly refer you to CQ2. Each per-trial network in Fig. 4F holds one stable fixed point at its cue angle, and these points lie on the teacher's ring. We have redrawn Fig. 4F per phase in the teacher's coordinates (App. F.3 of the revised paper), where both models share the same 150-unit space and need no alignment. In the stimulus, memory and response phases, JEDI's fixed points lie on the teacher's ring at the right cue angle (disparity 0.046, 0.028 and 0.101). In the context phase the teacher has a single central fixed point and no ring, and JEDI's points cluster near it. For harder examples, please see our answer to W3. Action: App. F.3 shows Fig. 4F redrawn per phase in the teacher's coordinates (see CQ2).

Thank you again for your constructive suggestions. The rank analysis and the mixed-structure teacher you proposed will be part of the revised paper. If our responses address your concerns, we would be grateful if you would consider updating your score.We are happy to answer any further questions.

id like to thank the authors for addressing my questions and their effort during the rebuttal period

We are glad the additional analyses were useful, and thank you for taking the time to review them. If the revisions have resolved your concerns, we would be grateful if you could reflect this in your updated assessment. Please let us know if any questions remain.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Common Questions (1/2)

Response to Common Questions

We thank the reviewers for their careful reading and constructive comments. Two concerns were shared across reviews, so we answer them here once. CQ1 covers whether condition-specific weights are needed, or whether inputs to a shared network could explain the same activity (wr71 W1/Q1, K3vT W1). CQ2 covers how well JEDI's MemoryPro fixed points recover the teacher's ring (2Krd Q2), and whether these fixed points behave like a ring attractor (wr71 W2/Q2). The remaining points will be answered in the individual responses.

CQ1. Are condition-specific weights needed, or could inputs to a shared network explain the same activity?

wr71 W1. The autonomous formulation cannot distinguish external drive ...
wr71 Q1. Please compare JEDI with a shared low-rank ...
K3vT W1. The experiments do not establish that varying ...

Thank you for raising this shared point. Recordings alone cannot fully separate a change in connectivity from an external input. Inputs from other areas and from unrecorded neurons are not measured. A model built on the recorded neurons, like JEDI, therefore folds them into the coupling between those neurons. Any trajectory r_c produced by the connectivity J_c that JEDI generates for condition c can also be produced by a shared connectivity J_0 plus an input u_c = (J_c − J_0)φ(r_c). For this reason, we read J_c as effective couplings rather than as the true wiring.

To see what these couplings capture, we used three synthetic teachers where the ground truth is known. In T-in, the connectivity is fixed and each condition gets its own sinusoidal input, as in our frequency-sweep teacher. T-w and T-mix are new. In T-w, the connectivity itself differs between conditions. In T-mix, each condition combines a limit cycle with one or two fixed points. The table below shows what JEDI recovers.

SettingWhat JEDI recovers
T-in, input not givenJEDI builds an internal model of the input. Its eigen-frequencies follow the input frequency (slope 1.03–1.28), as in Fig. 4A–B.
T-in, input given (τṙ = −r + J_c φ(r) + W_in u(t))The eigenpair at the input frequency disappears (slope −0.05 to 0.00). JEDI keeps the connectivity shared, as the teacher does.
T-w, T-mixJEDI recovers the teacher's eigenvalues (within 0.13–0.18) and its oscillation frequency in every condition and seed.
Monkey, target and phase as inputsJEDI still uses direction-specific weights (mean weights lower R² from 0.70 to 0.26), and both phases stay stable.

We also added the baseline that was kindly suggested, a shared low-rank RNN whose context sets a constant input, with matched parameters (260,213 vs 260,176). JEDI does better wherever the true connectivity differs between conditions, and the two models tie when only a known input differs. As with any model fitted to data, we view these spectra and stability properties as properties of the fitted dynamics. The table below gives the held-out R², where the shared-weight RNN is the context-input RNN, with the input added where it is known.

DataJEDI (Ours)Shared-weight RNN
T-w0.95−0.52
T-mix0.94−0.30
MemoryPro0.800.49
T-in, input given to both0.9940.996
MemoryPro, task inputs given to both0.850.58

Action. Section 2.1 describes J_c as effective couplings, Section 3.2 makes the internal-model interpretation of Fig. 4A–B explicit, and the context-input baseline, the input-aware fits and the new teachers are in App. F.1.

AuthorsOpenReview ↗

Response to Common Questions (2/2)

CQ2. Does Fig. 4F recover the teacher's ring, and do its fixed points behave like a ring attractor?

2Krd Q2. In Fig 4F it does not look ...
wr71 W2. Fixed-point geometry does not establish recovery of ...
wr71 Q2. Within a single inferred MemoryPro memory-phase network, ...

We fully agree that a ring attractor needs more than fixed-point geometry. In JEDI each cue angle has its own network, so the ring is traced by the fixed points as the angle moves around the circle. To address this, we characterised the fixed points, eigenvalues and eigenvectors of the MemoryPro networks as functions of the cue angle, with 36 cue angles and 3 seeds.

AnalysisWhat we measuredResult
Fixed points against cue angleAngle of each context's fixed point against its cue angleCircular correlation 0.99. Disparity to the GRU's fixed points 0.028 (p = 0.0002, permutation test)
Eigenvalues against cue angleSlowest eigenvalue of the Jacobian at each fixed pointNegative at all fixed points, between −1.00 and −0.21/τ, median −0.53 to −0.62/τ
Eigenvectors against cue angleOverlap of the slowest eigenvector with the ring tangent0.19 to 0.26 (random 0.05 to 0.06)
Perturbation responseReturn time after a small push along vs across the ringSimilar in both directions (ratio 0.85 to 1.11)

In the table, a circular correlation of 1 means each fixed point sits exactly at its cue angle. The disparity measures how far JEDI's fixed points are from the teacher's after rotation and scaling, from 0 (identical) to 1 (unrelated). The slowest eigenvalue gives how fast the slowest small perturbation decays, in units of 1/τ, and any negative value means the point is stable. The overlap is 1 when the slowest direction runs along the ring and close to 0 when it points across it. A return-time ratio near 1 means that a push along the ring and a push across it decay equally fast.

So each context holds one stable point at its angle, stability holds all around the ring, and together the fixed points form the teacher's ring. The angle is learned in the context, and each context's network returns to its point from any nearby state.

Taken together, these results show that JEDI recovers both the shape of the teacher's ring and the local stability of every point on it, with the cue angle carried by the context. We thank the reviewers for encouraging us to make this analysis explicit.

Action. App. F.3 reports the fixed points, eigenvalues and eigenvector overlaps against cue angle, and shows Fig. 4F redrawn per phase in the teacher's coordinates.

We thank the reviewers again for their time and feedback. We believe our responses address the concerns raised, and the revision substantially strengthens both the results and the clarity of the paper. As the discussion period ends soon, please let us know if there is anything else that needs clarifying. If our responses and the uploaded manuscript resolve the reviewers concerns, we kindly ask them to consider raising their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.