I think the work here is interesting, but I do not think this paper is ready for publication. The primary issue is a lack of comparison to existing methods and lack of discussion of prior work. For instance, this paper introduces an information decomposition to account for past history but does not discuss the various other ways that neuroscientists have thought about alternative information decompositions or alternative ways of regressing out trial history ("serial dependencies") to isolate different components of information. What specifically is novel or interesting about this work compared to previous approaches to information decomposition is unclear. There are many existing proposals for information decomposition out there; I sampled just a few in the references below, none of which appear to be cited or discussed here. Not all of these are directly relevant, but I hope the authors will do their own careful literature review and add a section comparing and contrasting their method to existing information decompositions, especially those that have been developed for a neuroscience audience.
- Throughout, terminology relating to Partial Information Decomposition (1) is used (e.g. "synergy" of information) but the origin of these terms is not cited, as far as I could tell.
- Ref (2) below discuss "intersection information" (II) to quantify the information in a neural population that is used to drive behavior. A point in favor of the present work is that (2) admits a weakness of II is that it does not account for context/history.
- Ref (3) below is a good source for information theory in neuroscience, and includes a discussion of partial information decompositions
- Ref (4) below discusses "unique" information, yet another information decomposition.
My second critique is related; the present study looks at decoding performance on a variety of existing neural datasets (DTU, EEGMMI). While this is an effective exercise in illustrating the mechanics of how the present information decomposition can be used to compute various bits of information, I was left wondering "so what?". There seems to be an unstated implication that prior attempts at looking at these datasets made some methodological or interpretive error, and that the present work corrects those past mistakes. If that's the case, this paper would be far stronger if the authors could show that past interpretations of the same data were missing something. Note: I am not simply saying that this is a benchmarking problem and that the authors need to "outperform" others on some benchmark. I am saying that the authors need to interpret the results and show us that their information decomposition tells us somehting about these data that was not previously known using other methods.
My third substantical critique is that the paper seems to imply that I[Y;X] can contain spurious information, and that accounting for C as in I[Y;X∣C] can better "attribute" information to X. This interpretation is not stated outright, but terms like "attribute" certainly suggest it, and it is not outright discussed or denied. The issue is one of correlation and causation. If one is interested, say, in the causal information in X that drives behavior Y, then it is unclear whether accounting for C is the right thing to do. After all, it could be the case that past trial history (C) is represented in the brain and measured by X, and this information has a causal role in determining Y. All this to say: "controlling for" context seems like a good idea at first, but what one should or should not control for in a given analysis depends on various causal assumptions (see, e.g. Pearl's "Book of Why"). The authors might not agree with my particular example here, which is fine. But I think there ought to be some discussion of how any measure of 'information' cannot answer causal questions, and therefore the application of these new information-decomposition tools might or might not be appropriate controls depending on the modeler's assumptions, the experimental setup, the particulars of X,Y, and C, etc.
Minor comments:
- formulae and numbers should be removed from the abstract; there is no context to make sense of them.
- Lines 43- are redundant with lines 106-
- many acronyms are used without definitions ("DTU", "EEGMMI")
- many terms are used before or without definitions ("calibration", "G", "fold map", the "L" and "H" subscripts in (8), "marginal correlation" and "conditional gain" in Fig 2, "R04" R08" and "R12", ...)
- The numeric bracket notation wasn't defined. Are the confidence intervals? Credible intervals? What percentile? How computed?
- In section 6.2, why account for just previous trial labels but not previous behaviors? It's common in the analysis of serial dependencies to consider the tuple (past label, past choice) or (past reward, past choice) to capture behaviors like 'win-stay/lose-shift'.
Thank you for the review.
Relation to earlier studies
The related-work section is revised so that each paragraph answers one question and ends with what our analysis adds:
Predictors and preprocessing
Section 5 names each model, and Appendix D lists every model and search grid in Table D.1, "Model and selection specifications".
DTU (auditory attention, 18 participants).
EEGMMI (motor imagery, 40 participants).
Context models. Logistic regressions on trial-history features, and a smoothed lookup table over the cue protocol.
Terms
All terms are now defined at first use. The ones you list mean:
What can be learned
Accuracy can miss what the log score measures. In auditory-attention EEG with contiguous folds, the previous label alone gives 0.945 balanced accuracy. Adding EEG raises it only to 0.948, yet the log-score gain is 0.041 [0.025, 0.057] bits per trial.
How much EEG adds depends on what the decoder already knows.