acceptodds
Under review as a conference paper at ICLR 2027

Image2MUA: Generative Modeling of Trial-level Macaque MUA with Area Structured Visual Expert Routing

Abstract

Understanding how visual stimuli shape neural population activity is a central goal of visual neuroscience. Visual encoding models predict mean responses to visual stimuli accurately, but mean responses alone do not capture response variability across repeated presentations of the same stimulus. Capturing these trial-level responses across cortical areas requires modeling correlated fluctuations while accommodating different visual feature preferences. We introduce IMAGE2MUA, a conditional rectified flow model that learns to generate joint multi-unit activity (MUA) in macaque primary visual cortex (V1), visual area V4, and inferior temporal (IT) cortex from one response per training image. IMAGE2MUA combines joint response generation with visual expert routing, which learns how individual channels draw on different visual representations. Across two macaques, its samples capture both trial to trial variability within individual channels and pooled residual correlations between channels, while their averages outperform the evaluated deterministic encoding baselines in mean response prediction. Learned visual expert preferences differ across cortical areas, even when area labels are not provided during training. Together, these results show that a single model can capture population variability across trials while making its use of visual representations explicit for each channel.

Primary area:
datasets and benchmarks
Keywords:
Neural encoding, Computational neural science, Multi-unit activity, Flow matching, Generative model, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_qbMHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Image2MUA models the conditional distribution of simultaneously recorded macaque V1/V4/IT multi-unit activity from visual stimuli. It uses rectified flow matching, population self-attention, and four frozen visual feature sources combined by a channel-specific router with a shared image-dependent adjustment. Training uses one response per image; repeated responses are reserved for channel-reliability screening and evaluation. On a class-disjoint split of TVSD from two macaques, the model improves average-response correlation over the evaluated deterministic baselines and captures marginal trial variability and pooled residual correlations. The paper also compares a matched-architecture diffusion formulation and tests whether area-associated routing structure persists when explicit area information is removed.

Critical strengths
  1. The distributional evaluation separates important aspects of success. Table 2 reports both CRPS and pooled noise-correlation similarity over five seeds. Collapsing samples to their own conditional mean worsens CRPS from 0.434 to 0.605 for Monkey F and from 0.400 to 0.553 for Monkey N. Channel-wise trial shuffling preserves the marginals while eliminating correlation similarity. These controls show that the model contributes more than a good mean or independently variable channels. The class-disjoint validation/test construction and matched-architecture DDPM comparison further strengthen the evidence. Appendix B also correctly treats the recorded-trial leave-one-out reference as neither an image-only baseline nor a formal bound.
  2. A useful extension to joint multi-area neural encoding, with strong mean predictions. One model generates a simultaneous response vector across V1, V4, and IT while retaining per-channel visual feature mixtures. On the class-disjoint split, overall mean-response correlations are approximately 0.87 and 0.89, compared with 0.84 and 0.75 for the evaluated TBE baseline (Table 1). The region-blind control removes both area inputs and area-balanced loss before recovering area-associated preferences, making the observed organization more informative than an analysis of a model directly supplied with area labels alone.
Critical weaknesses
  1. The incremental value over established or simpler probabilistic population models is insufficiently characterized. The principal generative comparator is a DDPM using the same architecture. Mean collapse and channel shuffling are useful diagnostic controls, but neither is a competitive alternative distribution model. The paper itself cites the flow/factor-analysis model of Bashiri et al. (2021), which already learns image-conditioned population distributions without repeated training presentations, but does not compare against that family. A strong shared-feature mean predictor with a regularized correlated-residual model would also be informative: pooled residual-correlation similarity can be high even when covariance does not depend on the image. Thus, the current results establish useful marginal distributions and pooled dependence, but leave uncertainty about how much benefit requires the proposed conditional generator. This limits the significance of the methodological advance, rather than invalidating the reported scores.
  2. The router is interpretable as a model mechanism, but its predictive necessity and stable area organization are only partially demonstrated. In Table 6, Global Static is better than the full router on Monkey F in both MCP (0.8746 versus 0.8708) and CRPS (0.4311 versus 0.4341); the full router improves both on Monkey N. Image-term interventions have very small effects, and channel-routing interventions barely change noise-correlation similarity (Table 3). Furthermore, the fully region-blind analysis uses one checkpoint per animal, whereas the five-seed no-area-embedding control retains the area-balanced loss and is a different experiment. Main/region-blind CLR correlation also retains global expert preferences, as the authors acknowledge. These results support an interesting association, but not yet a robust claim that the factorized router is essential to either distributional performance or reproducible cortical organization.
Questions for authors
  1. Assessment-changing comparison: How does Image2MUA compare with a correlated probabilistic readout using the same visual sources, or an adapted flow/factor-analysis population model, under the same single-presentation split? A comparison of MCP, CRPS, and pooled correlation error would clarify the generator's incremental value. Please also distinguish image-dependent covariance from pooled covariance if drawing conclusions about conditional population dependence.
  2. Assessment-changing control: Do the source-centered area preferences and area-associated variance remain consistent across independently trained fully region-blind models? Can a channel-only static router recover essentially the same results as the full channel-plus-image router? These controls would clarify which part of the routing design is consequential and which interpretation is reproducible.
  3. Clarification / evaluation uncertainty: Since the repeated test responses also determine the retained-channel set, how sensitive are the conclusions to the reliability threshold or to estimating reliability from a separate subset of repetitions? Please distinguish seed variability from uncertainty across held-out images. This would help quantify how strongly the reported gains depend on the evaluated sample rather than on optimization alone.
3 comments

Thank you for your thoughtful review, which strengthened our comparisons and reproducibility analyses. References are to the revised manuscript.

  1. Correlated probabilistic readout. Following your suggestion, we added Gaussian and Gaussian factor using the same visual sources and routing backbone under the same single-presentation split. Gaussian predicts independent channel distributions; Gaussian factor adds shared latent loadings. Training uses joint Gaussian likelihood (Appendix D.1, p. 18). Table 4 compares matched seeds 42–43; we also report CorrMAE. Means are reported; covariance MSEs are ×10⁻³.
MonkeyModelMCP ↑CRPS ↓NC ↑CorrMAE ↓AverageCov MSE ↓ImageCov MSE ↓
FGaussian.8527.4434.0002.08302.38561.2010
FGaussian factor.8714.4334.7728.0405.40741.1885
FImage2MUA.8719.4287.8125.0508.61541.1180
NGaussian.8709.4102−.0006.12662.9164.6647
NGaussian factor.8870.3998.7778.0607.4917.6588
NImage2MUA.8930.3986.7655.0825.9770.6269

We agree that Gaussian factor is a competitive generative framework, performing close to flow matching with strengths on different metrics. This does not undermine our central contribution: the routed cross-attention expert fusion design provides strong neural encoding and interpretable cortical associations; the competitive Gaussian factor baseline itself uses this same backbone with Gaussian output heads and shared latent loadings, demonstrating its encoding capability beyond flow matching.

  1. Average and stimulus dependent covariance. Following your suggestion, we added AverageCov MSE for average covariance error and ImageCov MSE for error in each image's deviation from that average (Section 4, p. 6). Both use 20 fixed 15/15 trial splits and cross-products of covariance errors to account for sampling noise.

  2. Retrained channel-only router. Channel (C) was retrained without image-routing logits (Table 2B, p. 8), achieving MCP/CRPS of .875/.429 in F and .889/.401 in N. C+I has lower mean ImageCov MSE in both animals: 1.1229 versus 1.1418 in F, and .6334 versus .6386 in N. Image-logit shuffling also produces small changes (Table 3, p. 10). These controls identify stable channel-specific feature selection as the more consequential routing component, with image reweighting contributing to stimulus specific covariance. All settings retain image features as expert inputs.

AuthorsOpenReview ↗

(Continued)

  1. Region-blind reproducibility. We expanded to three independent seeds per monkey (42–44), excluding area labels from inputs and loss. Figure 5 (p. 9) reports means and SD; Appendix Figure 7 (p. 25) shows every seed. It shows a stable preferences as below:

    Across both monkeys and all three seeds, V1 favors E0, V4 favors E1/E2, and IT strongly downweights E0. Source-centered profiles correlate across seeds at r = .879–.898 (F) and .802–.871 (N). Area-associated routing variance spans 15.38%–24.92% in F and 16.50%–29.52% in N. These are fully region-blind models, distinct from the no-area-embedding control.

  2. Image-sampling uncertainty and channel selection. We evaluated image-sampling variability separately from training-seed SD. For each monkey and main-model seed 42–45, we drew 50 image IDs with replacement from the 100 test images, in five independent batches of 1,000 resamples. Metrics were recomputed on matched draws and averaged over the four fixed seeds.

MetricF: all 100F: resampled 95% intervalN: all 100N: resampled 95% interval
MCP.8720[.8568, .8838].8915[.8719, .9049]
CRPS.4323[.4269, .4375].3998[.3934, .4062]
NC.8111[.6471, .7171].7719[.6217, .6975]
CorrMAE.0515[.0553, .0643].0815[.0754, .0983]
AverageCov MSE.6127[.4835, .8328].9638[.6425, 1.4564]
ImageCov MSE1.1251[.8638, 1.4309].6386[.3250, 1.0296]

Implanted channels may not all carry task-relevant visual responses. Higher thresholds favor reproducibility but may exclude useful, unstable signals; lower thresholds may admit task-irrelevant channels. Scores can vary with channel selection, but prediction accuracy alone cannot establish biological relevance. We therefore retained TVSD's mean pairwise reliability threshold >0.3, rather than exploring cutoffs based on model performance. Reliability averages over repetition pairs across 100 images with 30 presentations each. Repeats serve screening and evaluation; fitting and checkpoint selection use single presentations (Section 3.1; Appendix B, p. 14). The image-resampling analysis fixes this channel set; screening on separate repetitions was not tested.

Thank you again for your valuable feedback. We welcome further discussion.

Thank you for the response. I'll keep my score.

Reviewer_DN19OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(−1)
Read review
Paper summary

This paper introduces Image2MUA, an image-conditioned rectified flow model for jointly generating trial-level multi-unit activity across macaque V1, V4, and IT. The model combines population self-attention with four visual experts derived from frozen ResNet-18 and DINOv2 features. A factorized router combines channel-specific preferences with an image-dependent adjustment. Models are trained separately for two animals using one response per training image and evaluated against repeated responses to held-out images. The reported results show strong mean-response prediction, recovery of marginal variability and pooled residual correlations, and area-related expert preferences that remain visible without area labels.

Overall, I found the paper difficult to follow. The organization and explanations across sections could be clearer, and the main takeaways are not always easy to identify. Frequent cross-referencing between the text, equations, and figures adds to the reading burden and makes it difficult to develop a coherent understanding of the proposed approach, experimental results, and overall contribution.

Critical strengths
  • A relevant population-modeling objective. Jointly modeling trial-level responses across V1, V4, and IT is a useful extension of visual encoding beyond conditional means. Training on single presentations and evaluating against repeated responses to images from held-out classes provides a meaningful test of distributional generalization.

  • Strong reported predictive performance. Generated averages outperform the evaluated deterministic baselines in both animals and all three areas, with overall MCP values of approximately 0.87 and 0.89. Image2MUA also improves overall CRPS and noise-correlation similarity over the same-architecture DDPM–DDIM comparison.

Critical weaknesses
  • The additional architectural complexity is not sufficiently justified by the current comparisons and ablations.Image2MUA combines multiple visual sources, population attention, factorized routing, and an FFN MoE. Although the paper includes useful visual-source, routing, and area-information controls, it remains unclear which components account for the gains and what benefit the additional complexity provides. The deterministic baselines differ in visual features and readout architecture, so the mean-prediction improvements cannot be attributed specifically to distributional training. The same-architecture DDPM comparison is informative about the generative formulation, but retains the architectural components whose contributions remain uncertain. In particular, the benefit of the FFN MoE over a conventional FFN is not directly evaluated. A feature- and capacity-matched deterministic encoder for mean prediction, together with targeted simplifications within the generative model, would help distinguish the contributions of distributional training, feature selection, and architectural capacity.
  • The usefulness and reproducibility of the factorized routing mechanism are only partially supported. In Table 6, the full router improves MCP over Global Static by approximately 0.0036 in Monkey N but reduces it by approximately 0.0038 in Monkey F. Swapping the image-dependent routing term changes MCP by only −0.0014 and −0.0006, with little change in CRPS or noise-correlation similarity. The evidence for the added value of image-dependent routing is therefore limited.

Presentation weaknesses:

The presentation of this paper needs improvements for acceptance.

  • The method section is difficult to follow, especially Sections 3.1 and 3.2. The text in 3.2 is overly compressed, some sentences that are less technical but more intuitive can be helpful for understanding. Figure 1 is too complex and could be made more readable by indicating which boxes in panel a correspond to panels b and c. The locations of “channel image router” box are very confusing in both panels a and b. I am also unsure whether the different arrow colors and styles (dashed and solid) are meaningful and necessary. The use of arrows and colors in Figure 2 could also be improved. Some key transformations, such as the linear transformation, are not explicitly shown, requiring readers to move back and forth between the text and figures and making the method unnecessarily difficult to understand.
  • Equations 4, 5, and 6 requires extra efforts to understand. The presentation of the evaluation metrics could be simplified. Multiple averaging and correlation operations are nested within a single expression. Defining intermediate quantities, such as trial-averaged responses and residual correlation matrices, would make the computation easier to follow while preserving precision. Explicitly separating the computational steps would also help readers understand what each metric measures.
Questions for authors
  • Which improvements remain when visual features and readout capacity are controlled? What additional benefit does the FFN MoE provide over a conventional FFN?
  • How does a retrained channel-specific but image-independent router perform? Are the region-blind area–expert associations consistent across independently trained checkpoints?

Overall assessment

I currently lean toward rejection, primarily because the method description and Figures 1 and 2 are difficult to understand. This makes it challenging to reconstruct the architecture and confidently assess the contribution of its components. The authors need to improve the figures and provide a clearer, more coherent explanation of the method, particularly the data flow and the relationships between the attention and routing components. These presentation difficulties limit my confidence in the current assessment. I would be open to increasing my score if the authors address these issues and clarify the methodological questions raised in this review.

1 comment

Thank you for your thoughtful review and constructive suggestions, which helped us improve the presentation and strengthen the experimental comparisons. All references below are to the revised manuscript.

Depicting routed cross-attention experts at different depths of two pretrained networks was challenging, especially in Figure 1(b). Following your suggestions:

  1. Panel correspondence. The original colors, dashed boxes, and gradients did not convey the connections as intended. Panels (b) and (c) now exactly match the module names in panel (a): “Adapted Cross Attention” and “FFN MoE.” Matching icons link their inputs and outputs to the overview arrows (Figure 1, p. 3).

  2. Router and data flow. We repositioned the router and arrows in panel (b). Four image logits from ResNet-18/DINOv2 features combine with each channel’s four-dimensional logit vector to weight the experts for that channel and stimulus. This clarifies the routed visual pathway. All four visual experts are evaluated; top-2 selection applies only to the FFN MoE. Section 3.2 and the Figure 2 caption explain the projections, summary attention, and linear readout (pp. 3–4).

  3. Metric motivation. We supplemented the original computational details with metric motivations. Section 4 (p. 5) now opens:

    “We evaluate complementary... AverageCov MSE measures error in that average covariance, whereas ImageCov MSE measures error in each image’s deviation from it.”

    Each purpose is bolded before its calculation; NC separates centering, covariance averaging, normalization, and comparison (pp. 5–6). Section 5.2 reiterates: “This control isolates cross-channel trial alignment: the model’s high NC similarity captures dependence beyond marginal distribution fidelity” (p. 8).

  4. FFN MoE and readout capacity. Shared FFN removes the six routed FFNs and their router while retaining the shared FFN and visual pathway. We also trained large dense FFN, a single 256→7174→256 FFN matched to the full model's total trainable parameters, with the same visual features and training protocol. Both controls use seeds 42–43 per monkey. Full improves MCP and CRPS over both controls in both animals (Tables 4–5, pp. 17–18; regional results: Tables 6–8, pp. 19–21). Covariance MSEs below are ×10⁻³; Large denotes large dense FFN.

MetricF: SharedF: LargeF: OurN: SharedN: LargeN: Our
MCP ↑.8700.8693.8719.8905.8928.8930
CRPS ↓.4340.4341.4287.3988.4003.3986
NC ↑.8035.8151.8125.7660.7715.7655
AverageCov MSE ↓.6131.6023.6154.91721.0416.9770
ImageCov MSE ↓1.18311.14021.1180.6263.6214.6269
  1. Region-blind reproducibility. Area-associated preferences without area labels are a core contribution; we appreciate this suggestion. We expanded from one to three independent seeds per monkey (42–44), excluding area labels from model inputs and loss. Figure 5(a) (p. 9) now shows means and standard deviations; Figure 7 (Appendix G.1, p. 24) provides every seed’s bar plots and matched main/region-blind heatmaps. Across region-blind seeds, source-centered profiles correlate at r = .879–.898 (F) and .802–.871 (N).

  2. Matched deterministic baseline. Det-MSE retains the same visual features, response backbone, visual/FFN routing, and channel/area embeddings. It predicts means with MSE training, with response-state/time projection outputs zeroed (Appendix D.1, p. 17). Across matched seeds 42–43, Image2MUA improves MCP from .860 to .872 (F) and .865 to .893 (N). It also outperforms shared-backbone Gaussian and DDPM on MCP, CRPS, NC, and AverageCov MSE in both animals (Table 4, p. 17).

  3. Retrained channel-only router. Channel (C) was retrained without image-routing logits. Table 2B (p. 8) reports all available seeds: 2 for C, 5 for Static and C+I. Covariance MSEs are ×10⁻³.

MetricF: StaticF: CF: C+IN: StaticN: CN: C+I
MCP ↑.875.875.871.888.889.892
CRPS ↓.431.429.434.405.401.400
NC ↑.809.828.814.780.752.775
AverageCov MSE ↓.6770.5648.5981.87121.0219.9554
ImageCov MSE ↓1.12331.14181.1229.6407.6386.6334

C+I has the advantage on ImageCov MSE in both animals (Section 5.2, p. 9), while other metrics are not stable across subjects.

Thank you again for your valuable feedback. We welcome further discussion.

Reviewer_APB8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a generative approach to model the stimulus-conditioned distribution of multi-unit activity in the macaque visual cortex. The model relies on a rectified-flow formulation, with the primary modeling contribution being a mixture-of-experts (MoE) consisting of two layers of frozen DINOv2 and Resnet-18, respectively. The mixing weights are factorized into an image-dependent and channel-dependent contribution. The output of this backbone is then used to predict the local sample velocity governing the generative process. The model is tested on the TVSD dataset, which comprises simultaneous multi-unit recordings from areas V1, V4 and IT, therefore allowing for an explicit noise model across visual brain areas. The main claims regarding experimental results are

  1. The mean of generated samples captures recorded mean activity well, outperforming all tested deterministic models in terms of prediction-recording correlation.
  2. The marginal distribution of each channel is captured well as measured by the continuous ranked probability score (CRPS).
  3. The channel-channel covariance is captured well, as measured by the Pearson correlation between the covariance matrices of generated and recorded responses.
  4. Channel preferences with respect to the mixture of experts are systematically different across cortical regions, shown by studying the mixture probabilities as well as ablations.
Critical strengths

The correlation between mean predictions of the proposed model and recorded responses is high, and the model outperforms the baselines. While not the primary focus of this work, this is an important metric in the field and therefore as strength of the manuscript. The specific mixture-of-experts formulation proposed here is novel, to the best of my knowledge, and the strong performance on mean prediction could inspire future work in this direction. The noise correlations are strongly correlated with those found in the recordings, showing that the model clearly captures second-order statistics of the data distribution. A strong model of channel/channel-dependence can inform the search for latent mechanisms that account for trial-to-trial noise. The dataset and main experimental parameters are mostly suitable to support the analysis; claims are generally well-calibrated (see l.464ff for a good example).

Critical weaknesses

The clarity of the methods and parts of the results could be improved. The model description provides detailed equations, but misses a high-level, explicit description of how these computations are combined to obtain the overall model. A short paragraph in this direction could support Fig. 1. Further, I found the evaluations somewhat cumbersome to parse; brief statements on the intuition behind different metrics could make page 5 easier for the reader.

The motivation could be stated more clearly. The niche is established primarily in l.40f:

...a complete model must also capture the response distribution of each channel and how channels fluctuate together across trials.

I would argue that the modeling of fluctuations through a noise model does not necessarily make a model more complete, as the fluctuations may be caused by mechanisms that are not captured by the model rather than true stochasticity. Therefore, I would view the primary motivation of explicit noise models as an avenue towards identifying latent mechanisms that are not random, which can improve a model's capabilities (see Bashiri et al (2021), as cited in the text). As this avenue is not pursued in the manuscript at hand, I think a more specific argument for the work's motivation would strengthen it.

The distributional properties of the model are currently not evaluated sufficiently. The analysis considers only first- and second-order moments; beyond these summary statistics, a rigorous test for alignment with the target distribution is presented only for the marginal distribution of each channel (claim 2, see summary). Claim 2 also misses comparisons to simple parametric baselines (e.g. Gaussian, over/underdispersed Poisson, ...), meaning that the strength of the model on marginal distributions remains opaque. Without those, the CRPS score is difficult to interpret for the reader. Even if the model turns out to substantially outperform such baselines, the highly complicated parameterization of the model may make it difficult to extract concrete scientific insight from the marginal model.

The second-moment analysis (claim 3) is well-aligned with the scope of the work and the noise correlation in the data is clearly captured well. However, the manuscript currently misses an application of the model to a concrete scientific question. The subsequent analysis (primarily related to claim 4.) could also be performed with the deterministic baseline models. I believe that the work could be strengthened substantially by presenting a specific use case in which the advantages of Image2MUA over deterministic (or potentially weaker generative) models is demonstrated.

The analysis of visual routing is not yet in a state to derive definite conclusions from. While the analysis demonstrates mostly consistent patterns of preferences for both animals, they are currently not related to any specific properties of the different feature encoders. That some preference pattern exists is not necessarily surprising, so without additional analysis (e.g. wrt feature hiearchy/complexity, training objective, architectural details...) the insights from this section are somewhat limited. A similar criticism extends to the manuscript as a whole; I think the work could be strengthened by focusing less on individual scalar metrics and more on contextualization and interpretation of the results.

Overall, I think both the mixture-of-experts, as well as the generative approach show promise, but in the current form neither of the two are explored sufficiently to recommend the submission for acceptance.

Questions for authors

No specific questions.

3 comments

Thank you for your thoughtful review. We appreciate your recognition of the strong predictions, expert mixture formulation, and population dependence. Your suggestions helped us clarify the scientific interpretation. All references below are to the revised manuscript.

  1. Scientific motivation and conclusions. Our aim is to model the response variation observable when conditioning on an image. This includes variation arising from unobserved mechanisms; it does not require assuming that its biological causes are intrinsically random. The results support a shared visual readout across probabilistic formulations, with reproducible cortical organization in channel preferences and a smaller contribution from image dependent reweighting. Model comparisons, regional preferences, and routing controls support these conclusions.

  2. Simple probabilistic baselines and distributional interpretation. Following your suggestion, we added Gaussian and Gaussian factor models. Gaussian predicts independent channel means and variances; Gaussian factor adds shared latent loadings and is trained by joint Gaussian likelihood (Appendix D.1, p. 18). Table 4 compares the same visual and routing backbone using seeds 42–43. Entries are means; covariance MSEs are in units of ×10⁻³.

MonkeyModelMCP ↑CRPS ↓NC ↑AverageCov MSE ↓ImageCov MSE ↓
FGaussian.853.4434.00022.38561.2010
FDDPM.835.4608.77121.01201.1201
FImage2MUA.872.4287.8125.61541.1180
NGaussian.871.4102−.00062.9164.6647
NDDPM.830.4525.73311.5428.6184
NImage2MUA.893.3986.7655.9770.6269

The cross-attention readout combines different depths of ResNet-18 and DINOv2 and supports neural encoding under flow matching, diffusion, and Gaussian factor outputs. Image2MUA achieves the highest MCP and lowest CRPS in both monkeys. Gaussian factor captures average covariance especially well, while Image2MUA improves ImageCov MSE over it in both animals. These comparisons distinguish the ability to recover average population coupling from the ability to capture its image specific changes.

  1. Visual feature preferences and cortical organization. Following your suggestion, we connect routing preferences to the visual sources used by each expert. Appendix Figure 7 (Appendix G.1, p. 25) presents all three region-blind seeds per monkey (42–44), excluding area labels from model inputs and loss. Across both monkeys and all three seeds, V1 favors E0 (shallow convolutional features), V4 favors E1 and E2 (deeper ResNet and shallower DINO features), and IT strongly downweights E0. These preferences are measured relative to each expert's average allocation across all channels. Their reproducibility shows that channels in different cortical areas consistently favor different visual representations, even when cortical labels are excluded during training.

  2. Channel preferences and image dependent routing. Retrained channel-only models retain strong performance (Table 2B, p. 8). Shuffling image-routing logits changes MCP by −.0014/−.0006 and CRPS by +.0003/−.0001 in F/N. Shuffling channel preferences has larger effects on these metrics: MCP −.0075/−.0048 and CRPS +.0040/+.0041 (Table 3, p. 10). These controls identify stable channel-specific feature selection as the more consequential routing component.

  3. Model overview and metric motivation. We added an explicit overview connecting the computation stages (Section 3, Figure 1, p. 3). Section 4 (pp. 5–6) now explains each metric's purpose before its calculation: MCP measures mean tuning; CRPS evaluates marginal trial distributions; NC measures normalized pairwise dependence; AverageCov and ImageCov MSE distinguish average coupling from image specific deviations. Together, these evaluations connect stimulus tuning, trial variability, and population dependence within the same encoding framework.

Thank you again for your valuable feedback. We welcome further discussion.

Thank you for the interesting and detailed response.

I think the clarity of the metrics section is substantially improved in the revised version. The new model figure also helps with parsing the individual components more easily.

I also think that the additional analyses are quite interesting, even though they do not consistently support the work in the current form. In my view, the primary concerns I raised in the original review remain:

  1. The analysis of visual routing shows that image-dependent routing does not consistently improve results, but it makes the approach quite a bit more complicated. It seems to me like the final formulation should stick with input-agnostic routing. While this is not a big problem for future submissions, it would require substantial changes here that would deviate strongly from the originally submitted method.

  2. The comparisons to independent-channel Gaussians show a strong improvement in terms of AverageCovMSE (expected), but gains for ImageCovMSE are very small in comparison, indicating that second-order statistics are primarily captured at the average level. This makes image-dependent arguments very shaky. In terms of marginals, improvements over Gaussians are also very small; I can easily imagine that more realistic baselines, such as Poissons, would outperform the method on marginal distributions.

  3. I am still not fully convinced by the scientific motivation. Of course, I agree with the argument:

Our aim is to model the response variation observable when conditioning on an image. This includes variation arising from unobserved mechanisms; it does not require assuming that its biological causes are intrinsically random.

However, without explicit characterizations of these unobserved mechanisms I still don not clearly see the avenues opened up by this. I think this statement in the conclusion is interesting

Joint responsegeneration could complement these methods by simulating how candidate stimuli affect coordinated population variability, providing hypotheses for experimental testing.

but due to the seemingly limited stimulus-dependent noise modeling capacity, I am not convinced that this is realistic with the work at hand.

Overall, I would tend to retain the judgement that the paper is just below the acceptance threshold.

Thank you for acknowledging the improved clarity and additional analyses. We respectfully disagree with the broader conclusions drawn from the Gaussian comparisons. We believe these results should distinguish the contribution of the shared architecture from the strengths and limitations of individual probabilistic formulations.

1. Strong Gaussian results support, rather than negate, the shared architectural contribution.

Our methodological contribution centers on source-specific cross-attention experts and structured visual routing within a population-response architecture incorporating an FFN MoE. Flow matching provides an established generative formulation; we do not claim a new flow-matching algorithm.

Crucially, the Gaussian baselines retain our proposed visual expert and routing architecture. They are therefore alternative probabilistic formulations built on the same architectural contribution. Their strong performance supports the usefulness of this architecture across output formulations. It can limit the gains attributable specifically to flow matching, but does not establish that the underlying architecture is unnecessary or ineffective.

2. Explicitly learning shared covariance explains the Gaussian factor model’s AverageCov advantage, but does not establish equivalent conditional-distribution fidelity.

The Gaussian factor model explicitly learns a shared loading matrix LL, giving

$ \Sigma(x)=LL^\top+\operatorname{diag}v(x). $

Through joint Gaussian likelihood training, it directly parameterizes and fits a shared cross-channel covariance component. Its strong AverageCov performance is therefore consistent with a structural capability specifically suited to this aspect of the evaluation.

This model can relative accurately recover average coupling while remaining unable to express image-dependent off-diagonal covariance. Its success on AverageCov cannot, by itself, establish comparable performance on the conditional response distribution.

Table 4 illustrates this distinction. Gaussian factor achieves lower AverageCov error, whereas Image2MUA achieves lower ImageCov error in both animals: 1.1180 versus 1.1885 in F, and 0.6269 versus 0.6588 in N, in units of 10−310^{-3}. Image2MUA also has lower CRPS in both animals: .4287 versus .4334, and .3986 versus .3998, alongside higher MCP.

3. Limited image-routing gains warrant a narrower conclusion about that component.

We agree that the ablations do not establish image-dependent routing as essential. However, channel-only routing is a nested restriction of the proposed formulation, retaining the cross-attention experts, channel-specific mixtures, population backbone, and generative framework. Its competitive performance does not imply that the architecture must be substantially replaced. All routing variants also retain image-dependent visual features, so static routing weights should not be conflated with stimulus-independent response modeling.

4. The suggested superiority of Poisson baselines remains untested.

We disagree with using the imagined performance of an untested baseline as evidence against the method. Whether a Poisson-based model outperforms Image2MUA on marginal distributions is an empirical question.

Moreover, even if a Poisson output model built on our shared backbone achieved better marginal scores, this would demonstrate an advantage of that output formulation while retaining our cross-attention experts and routing architecture. It would qualify the marginal-performance advantage attributable to flow matching, rather than invalidate the architectural contribution. The existing Gaussian comparisons already illustrate why these contributions must be assessed separately.

We therefore ask that the assessment distinguish experimental results from hypotheses about untested alternatives, and distinguish the choice of output distribution from the contribution of the shared architecture.

Reviewer_eR8DOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces IMAGE2MUA, a conditional rectified flow model that generates joint, trial-level multi-unit activity across macaque V1, V4, and IT. The model combines population self-attention with channel-specific routing over frozen visual representations. It learns from one response per training image and is evaluated using repeated presentations of held-out images. The results demonstrate strong mean-response prediction, agreement with individual-channel response distributions, and recovery of pooled residual correlations between channels.

Critical strengths
  1. The emphasis on single-trial responses addresses an important limitation of visual encoding models. Trial averaging is useful for identifying stimulus selectivity, but it removes the variability and coordinated population fluctuations present during individual sensory experiences. A model can predict trial-averaged responses accurately while providing an incomplete account of neural population activity. Moving toward conditional distributions of simultaneous responses is therefore a scientifically meaningful objective. I view this emphasis as the paper’s central contribution.
  2. Learning response distributions from single-presentation training data is particularly valuable. Collecting many repetitions of every stimulus limits the diversity of images that can be studied experimentally. The ability to learn population variability from one presentation per training image, while evaluating against repeated responses to unseen images, makes this approach relevant to large-scale neural datasets. The class-disjoint evaluation further strengthens the generalization results. This contribution builds on earlier distributional encoding work and extends it to joint macaque V1/V4/IT responses.
  3. The evaluation distinguishes complementary aspects of predictive performance. IMAGE2MUA achieves mean-response correlations of approximately 0.87 and 0.89 in the two animals, outperforming the evaluated deterministic baselines. More importantly for the paper’s motivation, retaining the generated distributions improves CRPS by approximately 28% relative to collapsing those distributions to their means. Channel-wise trial shuffling preserves marginal distributions while eliminating residual-correlation similarity, providing a useful control that separates marginal fidelity from population dependence. The same-architecture diffusion comparison and five-seed results provide additional support.
  4. The routing analysis offers an interpretable connection to cortical organization. Area-related visual-expert preferences remain visible when cortical labels are excluded from both the model inputs and training loss. This is an interesting observation, although I regard it as supporting evidence rather than the primary reason for acceptance.
Critical weaknesses
  1. Strengthen the evaluation of joint and stimulus-dependent variability. The pooled residual-correlation results are encouraging, but pooling across images can obscure whether the model captures stimulus-specific changes in dependence. Reporting within-area and between-area correlation blocks separately, together with a measure of recorded-data reliability for these estimates, would help establish what aspects of multi-area variability are recovered. Comparisons with a simpler correlated-noise model or an established latent-variable population model would also clarify how much the proposed generator contributes beyond accurate means and flexible marginal distributions.
  2. Keep the routing conclusions proportional to the evidence. The full router does not consistently outperform Global Static across both animals, image-dependent routing interventions have small effects, and the region-blind analysis uses one checkpoint per animal. These findings support an association between learned feature preferences and cortical area, but provide weaker evidence that dynamic routing is essential for prediction. Additional region-blind seeds would strengthen this part of the paper. The use of repeated test responses for reliability-based channel screening should also remain prominent when describing the single-presentation setting. Potentially relevant literature The authors may find the following studies useful for contextualizing the importance and scope of single-trial modeling: • Xiao et al. (2024), “Feature-selective responses in macaque visual cortex follow eye movements during natural vision,” Nature Neuroscience. This study examines neural responses during natural image viewing and demonstrates the importance of gaze-dependent stimulus representations. It offers a relevant connection to extending IMAGE2MUA toward natural viewing, where eye movements change the effective visual input across individual fixations. • Hidalgo et al. (2026), “Trial-by-trial inter-areal interactions in visual cortex in the presence or absence of visual stimulation,” eLife, reviewed preprint. This work investigates single-trial predictability between cortical layers and areas, including shared variability during spontaneous activity. It provides a complementary perspective on the importance of population fluctuations and on the distinction between image-conditioned response generation and prediction informed by simultaneously observed neural activity. These are merely suggestions for discussion and context, rather than prerequisites for acceptance.
Questions for authors

I was surprised by the lack of any discussion about fixation and eye movements. Presumably all of the data involve fixating monkeys and fixation breaks were not included? The image to spike models often ignore these critical aspects of visual neuroscience.

1 comment

Thank you for your thoughtful review. Your suggestions strengthened our evaluation and neuroscience context, and we appreciate your valuable thoughts!

  1. Average and stimulus dependent covariance. Following your suggestion, we added AverageCov MSE and ImageCov MSE (Section 4, p. 6). AverageCov MSE measures error in covariance averaged across images. ImageCov MSE measures error in each image's deviation from that average, directly assessing whether the generator captures changes in population dependence with the stimulus.

    Tables 6–8 (pp. 20–22) report within area results; overall scores include both within and between area channel pairs.

  2. Generative model comparison. Following your suggestion for a simpler correlated population model, we added Gaussian factor, with a shared loading matrix and image dependent marginal variances. Table 4 (p. 18) compares the models using the same visual and routing backbone and training seeds 42 and 43. Entries are means; covariance MSEs are ×10⁻³.

MonkeyModelMCP ↑CRPS ↓NC ↑AverageCov MSE ↓ImageCov MSE ↓
FGaussian.853.4434.00022.38561.2010
FGaussian factor.871.4334.7728.40741.1885
FDDPM.835.4608.77121.01201.1201
FImage2MUA.872.4287.8125.61541.1180
NGaussian.871.4102−.00062.9164.6647
NGaussian factor.887.3998.7778.4917.6588
NDDPM.830.4525.73311.5428.6184
NImage2MUA.893.3986.7655.9770.6269

Image2MUA offers the strongest overall balance across these metrics, with the highest MCP and lowest CRPS in both monkeys. It also improves ImageCov MSE over Gaussian factor in both animals, capturing stimulus dependent covariance more accurately alongside better mean prediction and marginal distribution fidelity.

  1. Fixation and eye movements. We follow TVSD's fixation protocol (Papale et al., 2025, Neuron): eye position was monitored, and fixation within a 1°-diameter window was required throughout stimulus presentation and the following gray screen. We use only the released MUA from successfully completed presentations, preserving the original fixation screening.

  2. Reliability based channel selection. Implanted channels may not all carry task-relevant visual responses. Higher thresholds favor reproducibility but may exclude useful, unstable signals; lower thresholds may admit task-irrelevant channels. Scores can vary with channel selection, but prediction accuracy alone cannot establish biological relevance. We therefore retained TVSD's mean pairwise reliability threshold >0.3, rather than exploring cutoffs based on model performance. Reliability averages over repetition pairs across 100 images with 30 presentations each. Repeats serve screening and evaluation; fitting and checkpoint selection use single presentations (Section 3.1; Appendix B, p. 14).

  3. Region-blind reproducibility. Following your suggestion, we expanded the fully region-blind analysis to three independently trained seeds per monkey (42–44), excluding area labels from both model inputs and loss. Figure 5 (p. 9) reports across-seed means and SD; Appendix G.1, Figure 7 (p. 25) shows all six region-blind models and matched main-model profiles.

    Across both monkeys and all three seeds, V1 favors E0 (shallow convolutional features), V4 favors E1/E2 (deeper ResNet and shallower DINO features), and IT strongly downweights E0. Source-centered region-blind profiles correlate across seeds at r = .879–.898 in Monkey F and .802–.871 in Monkey N.

  4. Neuroscience context. We found your recommended studies very helpful. Xiao et al. (2024) connect gaze to visual representations during natural viewing. Hidalgo et al. (2026) address shared population fluctuations and prediction from simultaneously observed activity.

    We also explored stimulus synthesis and feature preferences in Ponce et al. (2019) and Bardon et al. (2022). These studies informed our discussion of visual preferences, population variability, and stimulus selection in the Introduction, Related Work, and Discussion (pp. 1–2 and 10).

Thank you again for your valuable feedback. We welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.