- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper builds a system that answers questions about 10-s clinical EEG segments. WA-Detector (5.14M parameters, self-supervised pretraining on about 1,170 h) finds nine waveform classes per channel and time and gets 45.22% mAP at tIoU 0.5 on 175 test windows from 26 patient-disjoint recordings (Table 1). In WA-Agent, a planner calls typed tools (metadata, background spectra, detection, sharp-transient measurement, plotting) and a responder answers from their structured outputs. WA-Bench tests selection, description and multi-turn conversation on the same 175 windows, using 1,550 AI-drafted, human-reviewed atomic rubric criteria that an LLM judge scores, plus a 30-case visualization task. WA-Agent reaches 93.71% on selection and 68.73% / 55.52% partial-credit rubric scores, well above image-only LLM baselines (Table 4).
Critical strengths
-
The benchmark keeps three kinds of correctness apart. Detection mAP, segment-level rubric coverage and tool routing get separate scores (Appendix A.8), and the rubric criteria are atomic, with strict and partial credit. The Oracle rows of Table 4 (detections replaced by annotations) are a good check: partial-credit description goes from 68.73% to 94.84%. So the detector is the bottleneck, not the language model.
-
The detector is studied closely and reported honestly. Tables 15-16 ablate pretraining stages, cross-channel modules, pyramid levels and augmentations. Figure 9 and Section 6.1.1 say plainly that every class is recall-limited, with about 60% of abnormal slow and fast events missed. If released, the annotated nine-class, channel-level dataset (28,634 training and 7,344 test events) would help other groups.
Critical weaknesses
-
Evaluation favours the agent and uses its own model as judge. The language tasks favour the agent by design. References, selection options and rubric criteria were drafted by an LLM from the structured annotation evidence (Appendix A.6, C.1) and checked by one medical trainee (L1009). WA-Agent's responder gets the same kind of structured class/channel/interval table from its tools. The image baselines have to read channel names and intervals off a plot (L407-409). The judge is DeepSeek-V4-Flash (L371-372), the same model the agent uses for both roles (L364). The sensitivity check swaps in DeepSeek-V4-Pro (Table 18), from the same family. Two results suggest the format matters. On selection, image-only GPT-5.5 already matches WA-Agent at 93.71% (Table 4). On CerebraGloss-Bench, which the authors did not build, WA-Agent is behind CerebraGloss on selection (70.00% vs 80.00%) and summary ROUGE-1. It is ahead only on the conversation score given by the same DeepSeek judge (4.27 vs 4.10, Table 34). The limitations paragraph admits that rubric scoring may miss unsupported statements (L480-481), and no expert rated the generated answers. I'd want a judge from another model family, an expert rating of a random sample of answers with judge-expert agreement, and a count of unsupported or hallucinated statements per answer. If possible, a second annotator should also write rubrics from the raw EEG, not from the structured evidence.
-
Test-set early stopping, single run. Appendix E.1.2 says the ablations "use the same training and test splits and test [email protected] for early stopping" (L1239-1240), and the main text says the variants share the full model's early-stopping rule (L356). Sadly, I could not find a validation split anywhere. The configuration lists one random seed (Table 14). The 45.22% headline, the 10.66-point gain over the adapted CerebraGloss-YOLO (Table 2) and ablation differences of 0.9-10.6 points (Tables 15-16) all come from test-selected checkpoints on 175 windows from 26 recordings of one hospital, and none has an interval. The agent results use this same checkpoint. The CerebraGloss-Bench detector comparison puts a new checkpoint against a published number under that paper's protocol (Appendix I.1). I'd ask for a patient-level validation split for early stopping and re-reported test numbers, with several seeds and bootstrap intervals over recordings.
Smaller points that don't decide my recommendation: the annotations come from one EEG specialist, checked by a medical trainee (L128-129), with no inter-rater agreement. This affects how to read both mAP and the rubrics. Selection uses "synthetic silver" answer keys (L370). The language results are from one run per configuration, with no repeated sampling.
Questions for authors
- How do the description and conversation scores, and the ranking against the image baselines, change with a judge from a different model family? How well does the judge agree with an EEG expert on a sample of answers?
- Was any validation split used to pick the WA-Detector checkpoint? If not, what is test mAP when early stopping uses a held-out set of training patients?
- What are the seed-to-seed spread and recording-level bootstrap intervals for the mAP values in Tables 1-3?
- Did the hospital data have ethics-board approval or a consent waiver? Does it cover the planned public release? And if the baselines were queried through third-party model APIs, does it cover sending EEG plots to them?
Response to Reviewer ciug
We thank the reviewer for the detailed feedback. We added cross-family rejudging, unsupported-statement counts, a text-only selection analysis, and validation-split, multi-seed, and confidence-interval analyses.
Response to W1 and Q1
Cross-family judge and expert check
GPT-5.6 Sol re-scored the frozen answers with the same rubrics. Partial-credit scores, DeepSeek/GPT (%):
Although GPT scores are higher, rankings of the seven systems agree closely (Spearman: description 1.000, conversation 0.893), with summary-priority and structured-only WA-Agent first and second under both judges. Under GPT, summary-priority WA-Agent leads the strongest image baseline, Gemini, by 27.88/22.98 description/conversation points (95% recording CIs [21.22, 31.90] and [12.74, 27.83]). Item-level agreement is substantial (78.11% exact agreement; linear-weighted kappa 0.7236), and an EEG expert reviewing a random sample of the judge's scores found no concerns.
Unsupported statements per answer
DeepSeek Flash extracted 74,400 claims from 6,160 answers and checked them against reference annotations and metadata. Per-answer counts and the partial-credit score after weighted penalties for contradicted and unsupported event claims, description/conversation:
WA-Agent reports far more supported events per answer, the fewest description contradictions, and the highest penalized scores on both tasks.
Selection performance of GPT-5.5
One reason GPT-5.5 approaches WA-Agent on selection is that the items are fairly easy: questions are direct and options rarely contain traps. We asked GPT-6.1 Sol to answer from the stem and options only, without EEG, and classified its rationales by cue sufficiency:
Candidate options thus let partial cues rule out distractors, explaining the high selection scores of most image baselines in Table 4. Open-ended answers require reporting waveform type and location directly, which better separates systems; this is where WA-Agent opens a clear gap.
Response to W2, Q2, and Q3
The submitted checkpoint was selected by test mAP without a validation split. We repeated the original protocol over four seeds and reran selection on four random patient-held-out splits:
Each split holds out six of 29 report-linked groups from the 34 training recordings; checkpoints are selected on validation and scored on the unchanged test windows. Even so, the mean mAP (40.31%) exceeds the test-selected CerebraGloss-YOLO (34.56%) and LaBraM (36.15%), a comparison favoring the baselines.
We report recording-level 95% CIs from 10,000 paired bootstrap draws over 26 test recordings; point estimates follow Tables 1-3:
Every ablation lowers mAP; the gain over CerebraGloss-YOLO and the contributions of pretraining and channel self-attention exclude zero, while the remaining ablations have wider intervals given the small test set. The CerebraGloss-Bench comparison uses a published score, so no interval is available.
The specialist's waveform annotations were checked by a medical trainee, who reported no major discrepancies.
Response to Q4 and Ethics Concern
Ethics review is pending. Only image-based evaluations sent EEG plots, stripped of all identifying information; other API calls used structured evidence and text.