- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on EEG analysis and decoding. Different EEG tasks may need different models and strategies, while traditional methods often use fixed pipelines or mainly rely on performance scores to guide the search. The authors propose NeuroAct, an agent framework with a dual-loop structure. Instead of just trying different methods and checking the final scores, it also uses intermediate steps like generating hypotheses, analyzing problems, and testing possible solutions. It also stores and updates experience from previous tasks to help with new ones. The experiments cover several EEG tasks and show good results.
Critical strengths
-
The experiments cover different EEG tasks and settings, with generally good results. The appendix also includes some detailed cases.
-
The idea of using intermediate analysis and hypothesis testing to guide research is interesting. I think it could also be useful for scientific problems beyond EEG.
Critical weaknesses
-
The method is mostly a combination of existing agent frameworks, EEG tools, and memory.
-
The diagnosis and hypothesis testing rely heavily on the LLM, and there is little evidence showing how reliable these intermediate judgments are, especially when they are wrong.
-
There is still a gap between the claims and the experimental evidence. The results show that the system works well, but do not clearly show that the gains come from the proposed dual-loop reasoning rather than the tools, domain knowledge, or search space.
Questions for authors
-
I am still not sure what the core methodological novelty is. The framework seems to rely a lot on the LLM's reasoning ability, domain knowledge, and tools. Since hypothesis generation and testing are central to this kind of research, I would like to understand what NeuroAct adds beyond these existing capabilities.
-
Following the first point, final performance is easy to evaluate with clear metrics, but how are intermediate hypotheses actually tested? What evidence is used to support or rule out a hypothesis? Is there any explicit updating rule, such as Bayesian updating, or is this mainly decided by the LLM? How is the reliability of these judgments checked?
-
Following the second point, how do the validation results affect future hypothesis generation? Is the feedback mainly kept in the prompt or research history, or is there a more explicit way to update the hypotheses being considered? How does the system decide which hypotheses to keep, reject, or explore next?
-
I did not see a comparison with a simpler feedback-driven agent using the same tools, domain knowledge, search space, and budget. How much of the improvement comes from the proposed dual-loop mechanism itself?
-
From the results, the evidence for experience reuse seems stronger than that for experience revision. How can we tell that revision itself helps? Are there repeated experiments showing consistent gains from revised experience?