- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CyberNeuro, a local-first workbench for automating neuroimaging workflows from natural-language requests. It integrates 67 processing and visualization skills with a planner, validator, dispatcher, and reporter, along with pinned execution environments, artifact checks, audit receipts, and an interactive quality-control interface. The authors also fine-tune a Qwen2.5-7B model, WandaMind, to make domain-specific tool-use and act-or-abstain decisions. They evaluate it on a 62-case tool-use/QC benchmark and assess the full system through anatomical DICOM-to-BIDS conversions from ten cohorts, three visualization/QC tasks, and offline-execution tests. The main reported results are an improvement from 40.3% to 72.6% on QC-62 for the full-precision model (62.9% for the deployed quantized model), completion of the ten conversion tasks without post-request intervention, reduced token use relative to NeuroClaw, and successful execution of the tested tasks under process-level network denial.
Critical strengths
- A relevant practical problem. Reducing the operational burden of neuroimaging analysis, especially when data must remain on institution-controlled hardware, is a worthwhile goal. The integration of local execution, existing domain tools, provenance records, and visual inspection could be useful to research groups without extensive infrastructure support. For this application, I do not view a lack of cross-domain generalization as an important limitation.
- Some evidence from actual execution. The paper goes beyond a purely text-based agent demonstration. The reported DICOM-to-BIDS runs produced outputs that passed the official BIDS validator, and the tested workflows ran with process-level network access denied. The failure-injection test also illustrates why checking actual output artifacts matters, rather than trusting an agent’s success message. These are useful feasibility results.
Critical weaknesses
-
Insufficient performance evaluation and attribution of the proposed agentic workflow. While domain-specific customization and integration are valuable, I believe that the most important criterion for evaluating an agentic workflow system is its actual performance. In this regard, I find the experimental evaluation insufficient in three key aspects. First, the evaluation provides limited evidence of generalization across different benchmarks and task settings, as the model-level results rely primarily on the relatively small QC-62 benchmark, while the end-to-end experiments focus on a narrow set of workflows. Second, the comparisons with existing approaches are not sufficiently fair or comprehensive. In particular, the comparison with NeuroClaw involves differences in the workloads and processing procedures, making it difficult to isolate the actual performance improvement. Third, the paper lacks sufficient ablation studies to identify the contributions of individual components, including the fine-tuned WandaMind model, the multi-agent orchestration architecture, and the validation mechanisms. Notably, the fine-tuned model shows no improvement in tool-selection accuracy and even a slight decrease in argument validity, with most of the reported improvement coming from abstention behavior. Consequently, although the proposed system appears practically useful, the current experiments do not convincingly establish how much improvement it achieves over existing methods or which of its proposed components are responsible for the gains.
-
The end-to-end evidence is narrow, and the principal efficiency comparison is confounded. The main real-data evaluation covers anatomical DICOM-to-BIDS conversion across ten cohorts, producing 61 in-scope NIfTI volumes. This is useful, but it does not test the longer processing and analysis workflows that motivate a 67-skill system. In the NeuroClaw comparison, the baselines do not perform the same amount of conversion work: NeuroClaw generates 188 volumes, including 127 outside the anatomical scope, whereas CyberNeuro generates only the 61 relevant volumes (Appendix E). The lower token consumption and wall-clock time may therefore partly reflect different conversion strategies and work scope, not just better orchestration. The visualization timing results are also difficult to interpret quantitatively because human setup and switching time is compared with CyberNeuro machine execution time (Table 2; Appendix F). Finally, four injected failures and thirteen clean conversions provide only limited evidence for general reliability, and passing a BIDS schema validator does not by itself establish the scientific correctness of downstream derivatives.
Questions for authors
- Matched comparison. Can the authors compare CyberNeuro and NeuroClaw using the same backend, in-scope series, target outputs, and conversion policy? A comparison against an established DICOM-to-BIDS automation tool would also help clarify the additional value of agentic orchestration.
- Contribution of the model and architecture. What happens if the unmodified 7B model, the deployed Q4 model, or a prompted/rule-constrained baseline is used in the same CyberNeuro workbench? Can the authors provide ablations for the fine-tuning stages and the main validation/orchestration components, preferably on complete tasks rather than only QC-62 first actions?
- Broader execution evidence. Can the authors report repeated end-to-end experiments for a small but representative set of longer workflows (e.g., reconstruction, registration, or connectivity analysis), including independent checks of scientific output quality and recovery from realistic failures? Such evidence would directly address the system’s main practical claim.
- Scoring clarification. Please reconcile the reported 100% abstention recall and 0% false-action rate with the per-item record for T35 in Appendix J, which shows a tool call and an overall failure for WandaMind. How exactly is T35 treated in the abstention calculation?
Rebuttal towards reviewer oASf
We thank the reviewer for the careful reading and for recognising the practical problem and the value of checking real output artifacts. Data and code: https://anonymous.4open.science/r/cyberneuro-artifacts-CD20/
W1. Evaluation breadth, tool selection and argument validity. A: We added 13 baselines on QC-62 (Qwen 7B–72B, with/without an act-or-abstain prompt, thinking on/off; none exceeds WandaMind's 72.6%, best 51.6%, Holm p<0.01), a 100-run workbench test (Q2) and per-session grades for 13 skills on five cohorts. On tool selection, the paper's 6-item subset hides a gain: over all 28 action items WandaMind picks the correct first tool on 64.3% vs 50.0%. We agree argument validity is slightly lower (87.5% vs 93.8%, n=16); malformed arguments are caught by the Validator's schema check before execution.
W2 & Q1. Matched comparison with NeuroClaw. A: Both systems got the same ten DICOM collections and the same goal (a BIDS dataset of the anatomical scans). The volume gap comes from design, not workload: NeuroClaw's agent runs one skill at a time, so it was asked to run dcm2nii and then bids-organizer; the first step converts every series before anything decides what is in scope (188 volumes, 127 out of scope); CyberNeuro classifies series from DICOM metadata first and writes the 61 in-scope volumes directly. Our normalization is conservative: tokens per NIfTI over NeuroClaw's own 188 volumes (per in-scope volume the gap would be 29.1×, not 9.4×), and wall-clock per dataset including stalls. With the same backend (DeepSeek V4 Pro for both), CyberNeuro uses 1.6× fewer tokens per NIfTI, 22.5% less wall time and 0 vs 24 interventions; WandaMind cuts tokens a further 5.8×. On timing in Table 2, we agree human and machine time differ (Appendix F); we will lead RQ3 with like-for-like interaction counts (applications 3–4 → 1, actions 14–20 → 1–2).
W2. Correctness beyond the BIDS validator, and reliability. A: dcm2bids output was compared with a lab-written reference image by image, including diffusion gradients, on five cohorts: with the Qwen-built skill it matched exactly on MCSA, ADNI, ADNIDOD and NIFD, and on 32/58 PPMI images; this check also caught wrong gradients that the validator passes. Beyond the 4+13 injected/clean runs, in 100 workbench runs host checks refused all 641 destructive-tool calls issued by the models.
Q2. Model and architecture contributions. A: In the workbench (untuned 7B vs deployed WandaMind Q4, 10 cohorts × 5 runs; no-model upper bound 10/10): every submitted task ran and produced BIDS; WandaMind loaded the Skill twice as often (8 vs 4), had 1 vs 47 turns stopped for repeated identical calls, and 225 vs 416 destructive-tool calls. Deployed Q4 on QC-62: 62.9%, abstention 100%, false action 0%. Prompted baselines: a rule raises abstention but lowers Overall (72B: 48.4% → 38.7%, over-refusal 7.4% → 22.2%). Fine-tuning stages (Q4 exports, same harness): untuned 26/62 → validated trajectories 34 → act-oriented augmentation 34 (escalation collapsed 4/4 → 1/4) → adversarial hardening 39 (escalation 4/4). Validation: without post-execution checks 4/4 malformed runs reported silent success (0/4 with them); pre-execution checks cut time-to-reject 112.2 s → 0.9 s.
Q3. Longer workflows. A: Registration and connectivity are part of CyberNeuro: its processing and QC skills (dcm2bids, neuroimage-pre, qc-agent) are the NeuroPilot pipeline (arXiv:2608.07541), run on 17 cohorts (>123,000 subjects): normalization via fMRIPrep/sMRIPrep and ANTs SyN; functional connectivity (fMRIPrep → XCP-D → nilearn, 19 parcellations) and structural connectivity (QSIPrep/QSIRecon); in infants 91% of SC and 98.4% of FC sessions usable. Independent QC checks: error injection on 500 ADNI subjects gave 100% specificity and 90.1–98.5% detection per step; five repeated QC passes on 200 subjects reached Fleiss' κ ≥ 0.82. For reconstruction, agents ran FreeSurfer and CIVET unattended on five cohorts against lab-written references (FreeSurfer: 268/270 measures identical). Long runs are supervised by scheduled check-ins that wake the agent until the job finishes or fails, and the LLM's role does not grow with run length: it routes each step, while computation and output checks are deterministic.
Q4. T35. A: WandaMind made no tool call on T35 and answered "Blocked: config+log write is not a derivative…", the intended abstention, so it counts as correct abstention and no false action. The Overall failure is a labelling error: T35's reference tool sequence was left as run_pipeline, while its target verdict and must-not list describe abstention. We will fix the label and report both versions (Overall 72.6% → 74.2%) and define abstention explicitly over T31–T36.
We hope these results address your concerns.