- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article explores the brain similarity of the model and proposes BrainLLM-Bench. By changing one factor (either the reading method, the audience, or the stimulus material) separately each time, it assesses the consistency of the model under different conditions. Through numerous experiments, it is shown that maintaining good stability after changing the audience is achieved, while changing the reading method may lead to a reversal.
Critical strengths
-
The author conducted a study by comparing the similarity between these language models and the human brain. This is an extremely interesting work. In the article, the author mainly discussed the significant differences between the reliability of rankings and the transfer of rankings, and conducted numerous experiments to verify this. Under different measurement conditions, this distinction was made clearer, and the reliability of the rankings was also evaluated.
-
This article also proposed cross-story experiments and analyses. Although it has not been confirmed whether they support the reversal, from these aspects, a broader interpretation can be supported.
Critical weaknesses
- It is somewhat questionable whether the comparison of reversed rankings between the two datasets is fair, as one method uses standardized features in Encoding, while the other RSA calculation relies more on raw geometric features. The authors need to thoroughly justify this difference and demonstrate whether it might affect the results.
- It should be considered that Encoding may be significantly influenced by regularization terms; thus, the authors need to conduct more comprehensive experimental analysis to verify whether the same conclusions as presented in the original paper still hold under optimal regularization settings.
- In the supplementary materials, after adding C4, the authors have not confirmed any ranking reversal. Therefore, they must provide a detailed explanation and analysis of how the inclusion of C4 alters these variables. Additionally, it remains unclear under what circumstances C1–C3 are applicable.
- The ranking reversal mainly occurs in the LibriBrain dataset of a single subject. The failure of cross-story transfer is currently an exploratory finding, but the applicable conditions for each item still need to be specified.
- It is suggested that the author provide a summary table, listing the data sets used for each analysis, the reading methods, and the control items, so that the relevant experimental settings can be more clearly understood when reading.
Questions for authors
The author should clarify whether the difference in choosing standardized features and original geometric features would have any impact? Would it lead to inconsistent interpretations of the results? The author regards these as the same brain similarity measures in the paper and suggests explaining the reasons for these two readings, as well as recommending the use of the same feature processing method for a sensitivity analysis experiment.
I will maintain my score.