- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that good cross-session decoding does not show a model has identified the same physical channels across sessions, and that matching channels within a session does not predict matching them across sessions. The main experiments compute features over raw neural activity, shuffle the channel identities, then solve the re-alignment problem. They find within-session recovery is high (though whole-map recovery is rare), cross-session recovery is near chance, and a baseline decoder is not affected by the errors in the recovered alignment. Follow-up experiments show that a permutation fitted with target labels can improve cross-session decoding without being stable across calibration splits (§5), and that a waveform-based unit matcher calibrated within one recording exceeds a prespecified false-match threshold when applied across sessions (§6). The paper concludes that identity recovery, decoder performance, and cross-session matching risk should be evaluated separately, and relates this to how POYO, NDT2, SPINT and LaBraM bind channels.The foremost weakness is that the paper is undigested AI output, and it is not possible to reproduce (or understand) the experiments from the text. AI use is acknowledged, but that is not an excuse, and there are indicators the author did not proofread.
Nearly every experiment is reported twice: a “frozen” version from an earlier iteration (presumably an earlier LLM session) and a “new” version with different settings and metrics (e.g. LINK 88.4% vs 86.3%, explicitly “not a revised estimate”; O’Doherty 4.7% vs 5.1%, “we retain both records”; Allen 20.7% retrieval vs 13.7% assignment recall with different denominators). §3 is titled “Analysis Status,” as if the work were unfinished. It is not explained what the new versions add beyond the old, and the duplication makes the paper difficult to understand.
It is not clear what model the primary findings (§4, Fig. 1) are evaluated under. The “fixed decoder” behind the headline 0.0018 R² loss is only inferable from an aside in §6 to be a one-bin ridge regression. The identity assay is described as “eight behavior-conditioned profiles from disjoint, behavior-balanced trial samples” with no statement of what the eight conditions are, what a profile contains, how balance is enforced, what distance is used, or what solver recovers the map.
Despite being framed against POYO, NDT2, SPINT and LaBraM (§7, Table 4), the results don’t say anything about whether the embeddings or performance of these models capture channel/neuron/unit identity. The experiments that shuffle activity channels and pair them back to their source are done on fingerprint statistics of the activity, not on any foundation model representation. The negative cross-session results show that these fingerprint statistics drift between sessions, a known property the authors themselves cite. I don’t think the results say anything about what a learned representation can or cannot identify, and I don’t think the work has implications for the design of the networks it references beyond what their multi-session mechanisms are already built to tolerate.
Critical strengths
The question of how permuting channel identity or perturbing activity affects the unit embeddings of POYO, or the read-in mechanisms of NDT2/LaBraM-style models, is interesting, and its answer could be mined for architectural improvements.
Critical weaknesses
The foremost weakness is that the paper is undigested AI output, and it is not possible to reproduce (or understand) the experiments from the text. AI use is acknowledged, but that is not an excuse, and there are indicators the author did not proofread.
Nearly every experiment is reported twice: a “frozen” version from an earlier iteration (presumably an earlier LLM session) and a “new” version with different settings and metrics (e.g. LINK 88.4% vs 86.3%, explicitly “not a revised estimate”; O’Doherty 4.7% vs 5.1%, “we retain both records”; Allen 20.7% retrieval vs 13.7% assignment recall with different denominators). §3 is titled “Analysis Status,” as if the work were unfinished. It is not explained what the new versions add beyond the old, and the duplication makes the paper difficult to understand.
It is not clear what model the primary findings (§4, Fig. 1) are evaluated under. The “fixed decoder” behind the headline 0.0018 R² loss is only inferable from an aside in §6 to be a one-bin ridge regression. The identity assay itself is described as “eight behavior-conditioned profiles from disjoint, behavior-balanced trial samples” with no statement of what the eight conditions are, what a profile contains, how balance is enforced, what distance is used, or what solver recovers the map.
Despite being framed against POYO, NDT2, SPINT and LaBraM (§7, Table 4), the results don’t say anything about whether the embeddings or performance of these models capture channel/neuron/unit identity. The experiments that shuffle activity channels and pair them back to their source are done on fingerprint statistics of the activity, not on any foundation model representation. The negative cross-session results show that these fingerprint statistics drift between sessions, a known property the authors themselves cite. I don’t think the results say anything about what a learned representation can or cannot identify, and I don’t think the work has implications for the design of the networks it references beyond what their multi-session mechanisms are already built to tolerate.
Questions for authors
No question could materially affect my assessment.