acceptodds
Under review as a conference paper at ICLR 2027

CyberNeuro: A Privacy-Preserving Agentic Workbench for Cohort-Scale Brain Imaging and Clinical Data Analysis

Abstract

Despite tremendous success in neuroimaging methodology, making large-scale, high-dimensional datasets ready for AI/ML applications remains a critical operational bottleneck. Conventional workflows require extensive manual effort across metadata curation, pipeline execution, post-processing quality control, and data management, a burden that disproportionately excludes laboratories with limited manpower and computational infrastructure. To address this real-world barrier, there is an urgent need for scalable, cost-effective computational platforms that democratize advanced neuroimaging analytics and accelerate discoveries in mental health and clinical translation. Capitalizing on multi-agent LLM breakthroughs, we introduce CyberNeuro, an agentic workbench with a tailored local LLM-model (called ‘WandaMind’) for automated neuroimaging and health-data analysis. Driven by four dedicated agents (Planner, Validator, Dispatcher, and Reporter) communicating via a secure MCP bridge and a pinned execution layer, CyberNeuro enables researchers to execute complex workflows using natural language; in the evaluated local configuration, all workflows completed with process-level network access denied. On the held-out QC-62 suite, WandaMind raises overall accuracy from 40% to 73% over its base model, mainly through safer abstention and fewer inappropriate tool calls; tool-selection accuracy is unchanged and argument validity drops slightly. Beyond automated metrics, the platform integrates a human-in-the-loop verification panel, which currently supports quality control of raw T1w images. On an end-to-end DICOM-to-BIDS suite of ten cohorts, CyberNeuro completed all tasks using about 10.6% of NeuroClaw’s per-volume token count with WandaMind and 61.7% with a cloud backend. The QC-62 benchmark and implementation details are provided in the appendix.

Primary area:
applications to neuroscience & cognitive science
Keywords:
AI agent, Neuroscience, LLM post training, Medical Image

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_oASfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CyberNeuro, a local-first workbench for automating neuroimaging workflows from natural-language requests. It integrates 67 processing and visualization skills with a planner, validator, dispatcher, and reporter, along with pinned execution environments, artifact checks, audit receipts, and an interactive quality-control interface. The authors also fine-tune a Qwen2.5-7B model, WandaMind, to make domain-specific tool-use and act-or-abstain decisions. They evaluate it on a 62-case tool-use/QC benchmark and assess the full system through anatomical DICOM-to-BIDS conversions from ten cohorts, three visualization/QC tasks, and offline-execution tests. The main reported results are an improvement from 40.3% to 72.6% on QC-62 for the full-precision model (62.9% for the deployed quantized model), completion of the ten conversion tasks without post-request intervention, reduced token use relative to NeuroClaw, and successful execution of the tested tasks under process-level network denial.

Critical strengths
  1. A relevant practical problem. Reducing the operational burden of neuroimaging analysis, especially when data must remain on institution-controlled hardware, is a worthwhile goal. The integration of local execution, existing domain tools, provenance records, and visual inspection could be useful to research groups without extensive infrastructure support. For this application, I do not view a lack of cross-domain generalization as an important limitation.
  2. Some evidence from actual execution. The paper goes beyond a purely text-based agent demonstration. The reported DICOM-to-BIDS runs produced outputs that passed the official BIDS validator, and the tested workflows ran with process-level network access denied. The failure-injection test also illustrates why checking actual output artifacts matters, rather than trusting an agent’s success message. These are useful feasibility results.
Critical weaknesses
  1. Insufficient performance evaluation and attribution of the proposed agentic workflow. While domain-specific customization and integration are valuable, I believe that the most important criterion for evaluating an agentic workflow system is its actual performance. In this regard, I find the experimental evaluation insufficient in three key aspects. First, the evaluation provides limited evidence of generalization across different benchmarks and task settings, as the model-level results rely primarily on the relatively small QC-62 benchmark, while the end-to-end experiments focus on a narrow set of workflows. Second, the comparisons with existing approaches are not sufficiently fair or comprehensive. In particular, the comparison with NeuroClaw involves differences in the workloads and processing procedures, making it difficult to isolate the actual performance improvement. Third, the paper lacks sufficient ablation studies to identify the contributions of individual components, including the fine-tuned WandaMind model, the multi-agent orchestration architecture, and the validation mechanisms. Notably, the fine-tuned model shows no improvement in tool-selection accuracy and even a slight decrease in argument validity, with most of the reported improvement coming from abstention behavior. Consequently, although the proposed system appears practically useful, the current experiments do not convincingly establish how much improvement it achieves over existing methods or which of its proposed components are responsible for the gains.

  2. The end-to-end evidence is narrow, and the principal efficiency comparison is confounded. The main real-data evaluation covers anatomical DICOM-to-BIDS conversion across ten cohorts, producing 61 in-scope NIfTI volumes. This is useful, but it does not test the longer processing and analysis workflows that motivate a 67-skill system. In the NeuroClaw comparison, the baselines do not perform the same amount of conversion work: NeuroClaw generates 188 volumes, including 127 outside the anatomical scope, whereas CyberNeuro generates only the 61 relevant volumes (Appendix E). The lower token consumption and wall-clock time may therefore partly reflect different conversion strategies and work scope, not just better orchestration. The visualization timing results are also difficult to interpret quantitatively because human setup and switching time is compared with CyberNeuro machine execution time (Table 2; Appendix F). Finally, four injected failures and thirteen clean conversions provide only limited evidence for general reliability, and passing a BIDS schema validator does not by itself establish the scientific correctness of downstream derivatives.

Questions for authors
  1. Matched comparison. Can the authors compare CyberNeuro and NeuroClaw using the same backend, in-scope series, target outputs, and conversion policy? A comparison against an established DICOM-to-BIDS automation tool would also help clarify the additional value of agentic orchestration.
  2. Contribution of the model and architecture. What happens if the unmodified 7B model, the deployed Q4 model, or a prompted/rule-constrained baseline is used in the same CyberNeuro workbench? Can the authors provide ablations for the fine-tuning stages and the main validation/orchestration components, preferably on complete tasks rather than only QC-62 first actions?
  3. Broader execution evidence. Can the authors report repeated end-to-end experiments for a small but representative set of longer workflows (e.g., reconstruction, registration, or connectivity analysis), including independent checks of scientific output quality and recovery from realistic failures? Such evidence would directly address the system’s main practical claim.
  4. Scoring clarification. Please reconcile the reported 100% abstention recall and 0% false-action rate with the per-item record for T35 in Appendix J, which shows a tool call and an overall failure for WandaMind. How exactly is T35 treated in the abstention calculation?
1 comment
AuthorsOpenReview ↗

Rebuttal towards reviewer oASf

We thank the reviewer for the careful reading and for recognising the practical problem and the value of checking real output artifacts. Data and code: https://anonymous.4open.science/r/cyberneuro-artifacts-CD20/

W1. Evaluation breadth, tool selection and argument validity. A: We added 13 baselines on QC-62 (Qwen 7B–72B, with/without an act-or-abstain prompt, thinking on/off; none exceeds WandaMind's 72.6%, best 51.6%, Holm p<0.01), a 100-run workbench test (Q2) and per-session grades for 13 skills on five cohorts. On tool selection, the paper's 6-item subset hides a gain: over all 28 action items WandaMind picks the correct first tool on 64.3% vs 50.0%. We agree argument validity is slightly lower (87.5% vs 93.8%, n=16); malformed arguments are caught by the Validator's schema check before execution.

W2 & Q1. Matched comparison with NeuroClaw. A: Both systems got the same ten DICOM collections and the same goal (a BIDS dataset of the anatomical scans). The volume gap comes from design, not workload: NeuroClaw's agent runs one skill at a time, so it was asked to run dcm2nii and then bids-organizer; the first step converts every series before anything decides what is in scope (188 volumes, 127 out of scope); CyberNeuro classifies series from DICOM metadata first and writes the 61 in-scope volumes directly. Our normalization is conservative: tokens per NIfTI over NeuroClaw's own 188 volumes (per in-scope volume the gap would be 29.1×, not 9.4×), and wall-clock per dataset including stalls. With the same backend (DeepSeek V4 Pro for both), CyberNeuro uses 1.6× fewer tokens per NIfTI, 22.5% less wall time and 0 vs 24 interventions; WandaMind cuts tokens a further 5.8×. On timing in Table 2, we agree human and machine time differ (Appendix F); we will lead RQ3 with like-for-like interaction counts (applications 3–4 → 1, actions 14–20 → 1–2).

W2. Correctness beyond the BIDS validator, and reliability. A: dcm2bids output was compared with a lab-written reference image by image, including diffusion gradients, on five cohorts: with the Qwen-built skill it matched exactly on MCSA, ADNI, ADNIDOD and NIFD, and on 32/58 PPMI images; this check also caught wrong gradients that the validator passes. Beyond the 4+13 injected/clean runs, in 100 workbench runs host checks refused all 641 destructive-tool calls issued by the models.

Q2. Model and architecture contributions. A: In the workbench (untuned 7B vs deployed WandaMind Q4, 10 cohorts × 5 runs; no-model upper bound 10/10): every submitted task ran and produced BIDS; WandaMind loaded the Skill twice as often (8 vs 4), had 1 vs 47 turns stopped for repeated identical calls, and 225 vs 416 destructive-tool calls. Deployed Q4 on QC-62: 62.9%, abstention 100%, false action 0%. Prompted baselines: a rule raises abstention but lowers Overall (72B: 48.4% → 38.7%, over-refusal 7.4% → 22.2%). Fine-tuning stages (Q4 exports, same harness): untuned 26/62 → validated trajectories 34 → act-oriented augmentation 34 (escalation collapsed 4/4 → 1/4) → adversarial hardening 39 (escalation 4/4). Validation: without post-execution checks 4/4 malformed runs reported silent success (0/4 with them); pre-execution checks cut time-to-reject 112.2 s → 0.9 s.

Q3. Longer workflows. A: Registration and connectivity are part of CyberNeuro: its processing and QC skills (dcm2bids, neuroimage-pre, qc-agent) are the NeuroPilot pipeline (arXiv:2608.07541), run on 17 cohorts (>123,000 subjects): normalization via fMRIPrep/sMRIPrep and ANTs SyN; functional connectivity (fMRIPrep → XCP-D → nilearn, 19 parcellations) and structural connectivity (QSIPrep/QSIRecon); in infants 91% of SC and 98.4% of FC sessions usable. Independent QC checks: error injection on 500 ADNI subjects gave 100% specificity and 90.1–98.5% detection per step; five repeated QC passes on 200 subjects reached Fleiss' κ ≥ 0.82. For reconstruction, agents ran FreeSurfer and CIVET unattended on five cohorts against lab-written references (FreeSurfer: 268/270 measures identical). Long runs are supervised by scheduled check-ins that wake the agent until the job finishes or fails, and the LLM's role does not grow with run length: it routes each step, while computation and output checks are deterministic.

Q4. T35. A: WandaMind made no tool call on T35 and answered "Blocked: config+log write is not a derivative…", the intended abstention, so it counts as correct abstention and no false action. The Overall failure is a labelling error: T35's reference tool sequence was left as run_pipeline, while its target verdict and must-not list describe abstention. We will fix the label and report both versions (Overall 72.6% → 74.2%) and define abstention explicitly over T31–T36.

We hope these results address your concerns.

Reviewer_3sk5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposed to use a local LLM with four agents (Planner, Validator, Dispatcher, and Reporter) to communicate over a secure MCP bridge, driving cohort-scale brain-imaging and clinical-data workflows via natural language.

Critical strengths

In this paper, the core concept is reasonable to address a challenging problem.

Critical weaknesses

The critical weaknesses concern the unjustified agentic design and the lack of ablations.

Questions for authors

Comments

  • What was the rationale behind the specific number design of employing four agents?
  • In this paper, the authors chose these specific roles (Planner, Validator, Dispatcher, and Reporter). What is the justification for this configuration, and why?
  • How does cyberNEURO ensure that researchers can effectively use natural language to execute complex workflows?
  • Given the inherent ambiguity of natural language, how is operational efficiency maintained?
  • Are there predefined prompts designed to guide researchers in executing these complex workflows?
  • In real-world scenarios, information is frequently incomplete, for instance, when a specific data modality is missing in a given case. How does the system handle such instances of missing information?
  • Skills and tools are critical to agent performance. Regarding skills, how was the total number of skills determined?
  • How does the system handle cases where a skill fails to function as intended or produces a negative outcome?
  • The authors could provide statistical analyses of the success and error rates for individual skills, as well as an evaluation of the model's efficiency in invoking these skills or tools.
  • In Section 2.7, both QLoRA and LoRA are used. What is the reasoning behind this choice? Would the experimental results be significantly affected if only one method (either LoRA or QLoR) were used consistently?
  • Some figures lack clarity and could be improved. For example, Figure C.1 contains an excessive amount of text and would benefit from further refinement.
3 comments
AuthorsOpenReview ↗

Rebuttal towards reviewer 3sk5

We thank the reviewer for finding the core concept reasonable. Data and code: https://anonymous.4open.science/r/cyberneuro-artifacts-CD20/

W. Justification of the agentic design and ablations. A: Only the Planner uses the LLM; Validator, Dispatcher and Reporter are deterministic host checks, justified by what each catches, which we now ablate:

  • Post-execution validation off: 4/4 injected malformed runs reported silent success; on: 4/4 blocked, 0/13 false rejections.
  • Pre-execution validation: median time to reject an invalid plan 112.2 s → 0.9 s.
  • Model swap in the same workbench (untuned 7B vs WandaMind, 10 cohorts × 5 runs each): host policy checks refused all 641 destructive-tool calls (416 untuned, 225 WandaMind); repeated identical calls had to be stopped in 47 vs 1 turns.
  • Training vs prompting: none of 13 new baselines up to 72B, prompted or not, reaches WandaMind on QC-62 (72.6% vs ≤ 51.6%).

Q1–Q2. Why four agents, and why these roles? A: The number follows the four trust boundaries between request and result; each role owns a failure mode the model cannot police itself: Planner (ambiguity: maps the request to registered skills, asks when a required input is missing), Validator (unsafe or malformed plans before execution; success claims without real output after), Dispatcher (environment drift: runs each skill in its pinned runtime and permissions), Reporter (untraceable results: hash-verified RunReceipts). Merging two would let a model statement replace a host check.

Q3 & Q5. Natural language for complex workflows; predefined prompts? A: No fixed templates. A workflow is a routing loop: each step the Planner routes to one registered skill with declared arguments (or asks, abstains, escalates); the Validator checks the call, the Dispatcher runs it, and the RunReceipt returns the verified outcome so the next step is routed from the actual result. Selecting a skill in the composer fixes the first step. WandaMind picks the correct first tool on 64.3% of QC-62's 28 action items vs 50.0% untuned.

Q4. Efficiency under ambiguity. A: Ambiguity is resolved before compute: clarifying questions, declared defaults, preflight rejection in 0.9 s. On RQ2, CyberNeuro used 10.6% of NeuroClaw's tokens per NIfTI with WandaMind, with 0 vs 24 manual interventions.

Q6. Missing information. A: Never filled silently: the Planner asks; input binding stops on a missing required input; a run without verified outputs yields an incomplete/failed receipt. On QC-62's missing-or-fake-result items, WandaMind abstains correctly on 6/6 (untuned 0/6, 27B 3/6). On real data, a tractography skill reported "No DWI data found" where cohorts keep DWI outside the main BIDS folders.

Q7. How was the number of skills determined? A: By coverage plus a validation bar, not a target. The 67 skills (32 processing, 35 visualization) cover every MRI data kind in an inventory of 33,326 files from mainstream cohorts; documented gaps drive new skills. The library grows through the execution-verified skill generator: on 13 tools, agents using its skills reached A/B against lab-written references on 10/13 tools, vs Paper2Agent 8, Skill Seekers 7, Code2MCP 1 (same model and limits).

Q8. Skill failures. A: Post-execution validation records completed/incomplete/failed; the Environment and Pipeline Doctors may only apply registered repairs, and every revised action is re-validated; skill versions are retained for replay. The generator delivers a skill only after its environment and entry points run on the target host; otherwise it refuses with evidence. Limitation we will state: a plan naming a wrapper instead of the tool can yield a "hollow" skill (3 of 5 heavy-tool skills from Nemotron; 0 of 12 from Qwen/Gemma).

Q9. Per-skill success rates and invocation efficiency. A: Visualization: all 169 callable units of the 29 visualization skills produce their declared output on real data with the network off. Processing: per-session grades for 13 tools × 5 models and on five cohorts (40 of 48 new-cohort sessions at A/B) are in the repository. Invocation: correct first tool 64.3% vs 50.0%; false action 0% vs 100% where the model must abstain.

Q10. QLoRA and LoRA. A: Only QLoRA is used: LoRA adapters (r=32, α=64) trained on a frozen 4-bit NF4 base; "LoRA" denotes the resulting adapter. Evaluation applies it to the bf16 base (72.6%); deployment merges it into 16-bit weights (merging into the 4-bit base corrupted the model) and quantizes to Q4_K_M (62.9%; abstention and false action unchanged). We will make this explicit in Sec. 2.7.

Q11. Figure C.1. A: Thank you. Figure C.1 is the appendix reference diagram for the generator; its per-stage text marks where model proposals end and deterministic verification begins, the point of the design, and the caption follows the same stages, so we would prefer to keep it.

We hope these answers address your concerns.

Dear Reviewer 3sk5, Thank you for your follow-up, we apologize again for late reply. A revised PDF is uploaded, the tables our first response only summarized are now in the paper and below.

1. "Four agents": corrected. Abstract, Fig. 1 and Sec. 2.3 now read: one LLM-driven Planner plus three deterministic stages (Validator, Dispatcher, Reporter).

2. Why these four stages: removing or replacing each one has a measured cost (Sec. 3.5, Tables 3–4).

StageWhat it checks, and whenEvidence when it is removed or replaced
Planner (trained)which skill and arguments, or ask, abstain, escalateprompt rule instead of training (Qwen2.5-72B): Overall 72.6 → 38.7, over-refusal 7.4 → 22.2; untuned 7B in the workbench: first call destructive 46/50 vs 18/50 runs, 47 vs 1 turns stopped for repeated identical calls
Validator, before executionthe plan only, before any computemedian time to reject an invalid plan 0.9 s → 112.2 s
Validator, after executionthe produced filesmalformed runs reported as success 0/4 → 4/4 (0/13 clean runs falsely rejected in both settings)
Dispatcherruntime and permissions while the tool runsthe correct call with no model at all produces BIDS on 10/10 cohorts; the host refused all 641 destructive calls issued by either model
Reporterwhat is recorded as the resultevery run ends in a hash-verified RunReceipt marked completed / incomplete / failed

The four stages act at different times (before, during, after the run, and at the record) on different objects (plan, running process, files, record). Merging two duplicates a check; dropping one leaves a failure undetected.

3. Per-skill statistics (Q9): Appendix C.2–C.3; full tables in Box 2.

per-skill success and error rates (Q9)

(a) Processing skills, end to end on real data (Appendix C.2, Table C.1). Unattended agent runs graded against hand-made, bit-reproducible lab references. A reproduced; B close (r ≥ 0.99, Dice ≥ 0.9, or within 0.1 SD); C deviates; D no usable output; E no skill. Cell: grade, verified builds of 3.

ToolDeepSeek V4.1 FlashQwen3.8-27BGemma 4 31BNemotron 3 120BLlama 4 Scout
MINDA 3/3A 3/3A 3/3A 3/3A 2/3
BrkRawA 3/3A 3/3A 3/3A 3/3A 2/3
PINGUA 3/3A 3/3D 3/3B 1/3D 3/3
CAPC 3/3A 3/3A 3/3A 3/3A 3/3
subnetrA 3/3A 3/3A 3/3A 3/3E 0/3
medmethodsB 3/3B 3/3A 3/3A 3/3B 1/3
neuroCombatC 3/3C 1/3C 3/3C 2/3E 1/3
neuroHarmonizeD 3/3C 3/3C 3/3D 3/3C 2/3
FSL BET+FASTB 3/3B 3/3B 3/3B 3/3E 0/3
SORDINOC 3/3C 3/3D 3/3C 3/3D 3/3
dcm2bidsC 3/3A 3/3A 3/3A 3/3C 3/3
FreeSurferA 3/3A 3/3A 3/3A 3/3E 0/3
CIVETA 1/3A 3/3A 3/3A 2/3E 0/3
Built (of 39)3737393520
A or B (of 13)8109104

Errors: C on harmonization tools and SORDINO comes from the agent's run-time analysis choices, not the environment; Llama 4 Scout fails at generation (20/39); 3 Nemotron heavy-tool skills were hollow (graded D).

(b) Reused unchanged on four further cohorts (Appendix C.2, Table C.2). n/m = images identical to the reference (incl. gradients).

ToolBuilt byMCSAADNIADNIDODNIFDPPMI
dcm2bidsDeepSeekC 56/64 (wrong gradients)C 16/24AAA
QwenAAAAC 32/58
GemmaAAC 18/20AC 31/58 (wrong gradients)
NemotronAAAAC 56/58
FreeSurferDeepSeek / NemotronAAAAA
Qwen / GemmaAABAA
CIVETDeepSeek / Qwen / GemmaAAAAA
NemotronAAD (hollow)D (hollow)D (hollow)

40/48 new-cohort sessions at A or B. Same model (Qwen3.8-27B), combinations at A or B (of 12): CyberNeuro 11, Paper2Agent 10, Skill Seekers 6, Code2MCP 3.

(c) Visualization skills (Appendix C.3, Table C.3). All callable units run on real data, network off; units passing / units:

afni-tractography-viewer 4/4brain-network-vis 4/4brainnet-viewer 6/6brainrender 5/5capcov 1/1circos 1/1
connectome-workbench 4/4dipy-horizon 4/4figures4papers 5/5freeview-browser 1/1fsleyes 3/3FURY 11/11
gephi 4/4ggseg 2/2layered-output 9/9Mango 3/3medmethods 1/1mne-source-viz 3/3
neuromarvl 4/4niivue 2/2nilearn 23/23paper-figures 7/7paraview 13/13pyvista 13/13
subnetr 5/5tavotto 1/1trackvis 10/10vedo 13/13visbrain 7/7total 169/169

(d) Invocation: 13 new QC-62 baselines (Appendix N, Table N.1). p: paired exact McNemar vs WandaMind 7B, Holm-adjusted.

ConditionOverallTool sel. (28)AbstentionFalse actionOver-refusalp
WandaMind 7B72.664.3100.00.07.4–
Qwen2.5-7B / +rule / +rule+3 shots41.9 / 40.3 / 38.753.6 / 53.6 / 46.40 / 33.3 / 66.7100 / 66.7 / 33.33.7 / 11.1 / 11.1≤ 0.0014
Qwen2.5-14B / +rule41.9 / 41.953.6 / 53.60 / 50.0100 / 50.03.7 / 7.4≤ 0.0017
Qwen3.6-27B bf16 / +rule / thinking51.6 / 43.5 / 38.746.4 / 39.3 / 50.066.7 / 83.3 / 033.3 / 16.7 / 66.711.1 / 14.8 / 11.1≤ 0.0082
Qwen3-32B / +rule / thinking41.9 / 48.4 / 45.250.0 / 57.1 / 53.650.0 / 83.3 / 16.750.0 / 16.7 / 83.37.4 / 7.4 / 7.4≤ 0.0078
Qwen2.5-72B / +rule48.4 / 38.757.1 / 39.350.0 / 83.333.3 / 16.77.4 / 22.2≤ 0.0082

(e) Training rounds (new; Q4_K_M exports, same harness), QC-62 items passed:

UntunedRound 1: validated coreRound 2: act-orientedRound 3: adversarial hardening
Overall (of 62)26343439
Escalation (of 4)0414
Command construction (of 6)0033
Quality judgment (of 11)0676
Hard-gate failures6141

Round 2 adds action but collapses escalation; round 3 (21 mined counterexamples) restores it. Intermediate rounds scored once.

If our response has adequately addressed your concerns, we would sincerely appreciate it if you could reconsider your score.

Thank you once again for your time, patience, and thoughtful feedback. We sincerely apologize for the delayed response.

We wish you a wonderful day!

Best regards,
The Authors

Reviewer_7vHROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

CyberNeuro is a local first desktop workflow and app for executing neuroimaging workflows through natural language. The main contribution is the integration of data processing and visualization agentic skills which are used through a four stage architecture where the LLM routes the data flow through deterministic processing, validation and reporting blocks. With this the authors describe WandaMind, a fine-tuned model on domain specific trajectories for which the authors report baselines on their own unreleased benchmarks.

I couldn't find any supplementary material including: code, benchmark datasets, the application, fine-tuned model weights, trajectories; to verify any of the claims made in this paper.

UPDATE: The authors provided the code which on a glance seems to fit the paper. As it was last day of the discussion period I had no time to go into depth. The authors also provided the fine-tuned model weights, however I also did not evaluate those for the same reason.

In its current form, the manuscript reads as a description of software functionality and software engineering rather than a research contribution.

Critical strengths

A potential contribution of the paper is the focus on locally deployable agents for neuroimaging data for preserving privacy. However this potential is fully unrealized. What would be required is - releasing the fine tuned weights and the evaluation artifacts; readjusting the focus of the paper from the application to model adaptation and demonstrating robust improvements over strong baselines. Releasing, or constructing the required benchmark datasets so that community could improve these locally deployable agents further.

UPDATES: The authors released the weights and the evaluation artifacts but as it is too late in the review period I am unable to review those in depth.

Critical weaknesses
  1. The methodological novelty is not clearly established. The paper primarily integrates existing techniques, with extremely limited ablation testing, which leaves us in the dark to whether it's even a good integration of the existing techniques (fine-tuning, hierarchical workflows, etc.). Several architectural features that the authors introduce are already described in prior work, which is cited, but there is no meaningful contribution that makes CyberNeuro a publishable improvement.
Questions for authors
  1. Can you provide any of the supplementary material that would allow me to verify the claims made in this paper?

  2. What evidence establishes the benefit of the four stage architecture, or the necessity of any of the stage over another, over simpler methods such as just better prompting?

  3. Are there any additional tests comparing stronger baselines to WandaMind, and what made you limit the baseline comparisons to a single 27B as the upper range?

4 comments
AuthorsOpenReview ↗

Rebuttal towards reviewer 7vHR

We thank the reviewer for the detailed review and for pointing to locally deployable, privacy-preserving agents as the potential contribution. We agree that verifiable artifacts and strong baselines are essential, and we address both below.

W1 & Q1. Supplementary material to verify the claims. A: Yes. The following is now available in an anonymous repository: https://anonymous.4open.science/r/cyberneuro-artifacts-CD20/

  • QC-62: all 62 items (prompt, category, reference tool sequence, key arguments, must-not behaviours), system prompt and tool schema;
  • scorers, per-item records of every model (paper and rebuttal) and the evaluation harness, which reruns QC-62 on any OpenAI-compatible endpoint, so no baseline needs to be taken on our word;
  • the deployed WandaMind Q4_K_M weights (linked from the README, with their sha256);
  • the system-level experiment below: driver and per-run logs (tool calls, replies, receipts) of all 100 workbench runs;
  • for the skill generator: the grade of every agent session (13 tools x 5 local models x 3 generations), the comparison with three existing generators, and the reuse on five cohorts.

Subject-level images stay under the cohorts' data-use agreements (ADNI, PPMI, etc.). As a reproducibility check, rerunning the untuned 7B under a different vLLM version gives 26/62 vs 25/62 with the identical first tool call on 56/62 items, and our script reproduces Table 1 and its McNemar p-values from the per-item records.

W2. Novelty and ablations. A: We agree that the ingredients (fine-tuning, staged workflows) exist; our claim is narrower. The contribution is (i) a small local model trained to act, abstain or escalate under tool use, and (ii) a host that never accepts a model's claim as evidence, plus (iii) an evaluation that separates the model's decision from execution. The rebuttal adds the ablations needed to test each part:

  • Training vs prompting (13 new conditions, Q3): no prompted or larger model reaches WandaMind.
  • Model swap inside the system (same workbench, 50 runs per model): replacing WandaMind with the untuned 7B leaves execution intact but changes the model's failures: 47 vs 1 turns stopped for repeated identical calls, 416 vs 225 calls to a destructive tool.
  • Post-execution validation on/off: silent success in 4/4 injected malformed runs without it, 0/4 with it, and 0/13 false rejections.
  • Pre-execution validation on/off: median time to reject an invalid plan 112.2 s → 0.9 s.

Q2. Benefit of the four stages over better prompting. A: Only the Planner is model-driven; validation, dispatch and reporting are deterministic host components. They are therefore not alternatives to prompting but checks that hold whatever the prompt says, and each covers a failure no prompt can rule out:

  • Validator (pre-execution) rejects malformed or unsafe plans before compute is spent (112.2 s → 0.9 s).
  • Validator (post-execution) turns success claims into checks on real files: without it, 4/4 malformed runs reported silent success.
  • Policy checks hold against the model: in our workbench runs, the untuned 7B, given the same system prompt, made 416 calls to a destructive tool (77 aimed at input data) and had 47 turns stopped for repeated identical calls; host checks refused every one, and nothing was deleted.
  • Dispatcher runs each skill in its pinned runtime: issuing the correct pipeline call without any model produced BIDS on 10/10 cohorts, so execution never depends on model output.
  • Reporter writes a hash-verified RunReceipt for every run, so every claim can be audited afterwards.

Q3. Stronger baselines beyond 27B. A: The 27B model was the largest that runs on a single local workstation GPU, our deployment target. In response we added larger and newer models under the identical harness (temperature 0, released scorer unchanged), QC-62 Overall (%):

ModelPlain+ act-or-abstain ruleThinking
WandaMind 7B (ours)72.6––
Qwen2.5-7B (untuned)41.940.3 (+3 shots: 38.7)–
Qwen2.5-14B41.941.9–
Qwen3.6-27B (bf16)51.643.538.7
Qwen3-32B41.948.445.2
Qwen2.5-72B48.438.7–

None of the 13 new conditions exceeds WandaMind; every difference stays significant after Holm correction (paired exact McNemar, p<0.01). Scale and prompting raise abstention but not Overall: with the rule, Qwen2.5-72B's abstention rises 50% → 83%, but its over-refusal on items that require action rises 7.4% → 22.2%. Thinking mode does not help either: reasoning models follow "discover skills before running one" literally and search even on requests they should refuse.

We hope the released artifacts, new baselines and ablations address your concerns, and we would be grateful if you would reconsider your assessment.

As this is the last day of the review period I am unable to go too much into depth in the provided code. I credit you for providing it along with model weights and I wish you did it sooner so I could've validated it myself. This gives the paper a lot more credibility, but I still feel the rebuttal falls flat on following points.

Regarding Q2 I was wondering if the behavior could be achieved through better prompting disregarding your deterministic approach.

Regarding Q3 I wanted to see baselines compared to some other frontier models. You can also have privacy on cloud deployment with larger models. It would not have taken away from your score since I understand the deployment target. I'm just not sure if it's than a meaningful scientific contribution.

Dear Reviewer 7vHR

We are very sorry that the code and model weights came so late. Over the discussion period, we tried to run every additional experiment the reviewers asked for, and we wanted to release everything together in a form anyone could rerun. That took us longer than we had hoped. We fully understand that this left you very little time to check it, and we truly appreciate that you still took a look and gave us credit for it. Below we reply to your two remaining points.

On Q2: could better prompting alone achieve this, without the deterministic parts?

A: Thank you for clarifying the question. QC-62 already measures exactly this setting. It scores the model by itself, with no Validator, Dispatcher or other host check involved, so every QC-62 number reflects only the model and its prompt. To give prompting a fair chance, we wrote two stronger prompts. The first adds an explicit act-or-abstain rule to the system prompt, say “blocked” when a result is missing or incomplete, escalate anything beyond the local model, ask one question when information is missing, and otherwise search and then run. The second adds three hand-written examples to the rule (a blocked result, an escalation and a normal action). None of them is taken from QC-62, their token-set Jaccard similarity to any QC-62 prompt is at most 0.42.

The rule did make models abstain more, but they also began refusing requests they should have carried out. For Qwen2.5-72B, abstention went from 50% to 83% and over-refusal from 7.4% to 22.2%, while Overall dropped from 48.4% to 38.7%. With the three examples, the untuned 7B scored 38.7%. The best prompted result from 7B to 72B is 51.6%, WandaMind reaches 72.6%. Our reading is that a prompt can tell a model that refusing is sometimes right, but the model still has to learn which cases call for it, and that is what fine-tuning taught WandaMind.

Prompting also had clear limits inside the running system. All 100 workbench runs used the same system prompt. Even so, the untuned 7B called a destructive tool 416 times and was stopped 47 times for repeating an identical call. WandaMind, trained and prompted, still made 225 such calls, and in two runs it said it was starting the pipeline without ever calling it. For this reason the system never treats what the model says as evidence. The host refused all 641 destructive calls, and in the failure-injection test it caught all 4 malformed runs that the model had reported as successful. However, we have not yet run one ablation you mentioned, i.e., a single agent with every check removed, relying on the prompt alone, evaluated on complete tasks. We will add it to the final version. Thank you so much for this helpful suggestion.

On Q3: comparison with frontier models

A: CyberNeuro is built around a local model on purpose. Many hospitals and medical schools do not allow clinical data to be sent to cloud AI services at all, so for these sites a cloud model, however strong, is not an option. Many smaller labs Running on the institution’s own workstation is what lets both groups use the system. For this reason we compared WandaMind against open models that can be deployed locally, going up to Qwen2.5-72B.also lack the infrastructure or budget for large API workloads. That model is about ten times the size of WandaMind and already needs more than a single workstation GPU. No condition up to that size reaches WandaMind on QC-62.

We also ran a much larger open model at the system level. DeepSeek V4 Pro is open-weight, and we served it on our own server. With it as the backend, CyberNeuro completed the same ten-cohort DICOM-to-BIDS task, and NeuroClaw ran on the same DeepSeek V4 Pro, so the comparison between the two systems holds the backend fixed. WandaMind completed the same task with 10.6% of NeuroClaw’s tokens per volume, against 61.7% for DeepSeek V4 Pro.

In addition, a larger model in a private cloud can also protect privacy where an institution permits it, and closed frontier models would be a useful reference. We see the scientific contribution in the other direction: showing that a small model, trained on the right data, can outperform models ten times its size on this decision. That is what makes a reliable agentic workbench possible on hardware a hospital can actually use.

On the scientific contribution

A: CyberNeuro itself is a contribution, to our knowledge, it is the first privacy-preserving agentic workbench that runs cohort-scale neuroimaging analysis entirely on local hardware. Saving researchers time was the main reason we built it, and it is now used at more than 20 labs, where it has saved researchers a great deal of time and received largely positive feedback. Beyond the system, we make three further contributions.

Specifically, first, scale and prompting did not produce calibrated abstention. Across 15 baseline configurations from 7B to 72B, including prompt variants and thinking mode, no model abstained on every case that required it, and the best overall score was 51.6%. After targeted post-training, our 7B model reached 72.6% overall (p≤0.004p ≤ 0.004 against every baseline), abstained on all such cases with no false actions, and kept the same over-refusal rate as the base model (7.4%). A 4-bit quantized version retained 100% abstention at 62.9% overall. Second, we keep LLM routing separate from deterministic verification, so each error can be traced to either the decision or the execution step. In our runs, the verifier caught all 4 silent errors that occurred. Third, we release QC-62, the evaluation harness and the WandaMind weights, so others can run controlled experiments on local scientific agents. We are continuing to improve the system based on this feedback.

Thank you very much for taking the time to engage in this discussion, and we hope our response has addressed your concerns.

Have a nice day.

Reviewer_2mvKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is interesting, but it seems more focused on software development using LLMs in an agentic workflow. CyberNeuro automates the preparation of MRI data for analysis. It consists of using tools such as FreeSurfer / fMRIPrep. LLMs turn the request into tasks built from registered skills. A deterministic check is put in place to make sure that input/ permissions are all sound. After the CyberNeuro system runs the tools, it checks the outputs to ensure they exist and are valid. The software system runs the tool and gathers the outputs into reports.

Critical strengths

The paper addresses a real problem with a nice good design. It proposes using a local LLM model which works well with institutional and privacy constraints. This looks like a reproducible piece of work to me, the authors provide all necessary details behind the CyberNeuro system they developed. Not that is is showstopper but the contribution does fall more into a contribution associated with software engineering rather than new machine learning methods or datasets or evaluation, etc, I am not sure it is a strong fit for a venue like ICLR but I enjoyed reading it.

Critical weaknesses

I think novelty is on the weaker side. Probably unfair to just say it is more on the integration of LLMs and tools in neuroimaging. Which I think is a great area – harnessing brain imaging in research. The result of fine-tuning improving abstention only rests just on the 62-item benchmark the authors built themselves. It would have been nice to see other open sourced benchmarks but I not too familiar with this specific area on imaging and the availability of such publicly available benchmarks. The paper compares a human’s time (which includes time spent learning the tools) with machine execution time. I am not fully convinced this is a fair comparison. I think this is a solid paper for an ML systems or AI dev conference but not sure about its fit in a venue like ICLR.

Questions for authors

Could the authors also provide the results with an untuned 7B model inside the CyberNeuro system?

1 comment
AuthorsOpenReview ↗

Rebuttal towards reviewer 2mvK

We thank the reviewer for the clear summary and for noting that CyberNeuro "addresses a real problem with a nice good design", suits privacy constraints, and is reproducible.

W1. On fit with ICLR and the ML contribution. A: Thank you. Part of CyberNeuro is systems work, which ICLR's call for papers includes, as it does applications to neuroscience. Beyond integration, we make three ML contributions:

  1. A learned act/abstain/escalate policy for a small local model. WandaMind (7B) reaches 72.6% on QC-62, above all 13 new baselines up to 72B (best 51.6%; W2). Prompting cannot replace it: an act-or-abstain rule lowers Qwen2.5-72B from 48.4% to 38.7% and raises its over-refusal from 7.4% to 22.2%.
  2. LLM-routed, tool-deterministic execution. A model's success message is never taken as evidence: without post-execution validation all 4 injected malformed runs reported silent success; with it all 4 were blocked, with 0 of 13 false rejections.
  3. An evaluation separating the model's decision from execution: QC-62 scores tool choice, arguments, abstention and false action separately; our new system test (Q1) adds a no-model upper bound.

We will state them in the introduction.

W2. On the abstention result resting on our own benchmark. A: A fair concern; we answer it in three ways.

  • Leakage control and release. Max token-set Jaccard similarity between any training candidate and any QC-62 item is 0.55; template families stay in one split. We release all items, scorers, prompt, tool schema, per-item records and the deployed WandaMind weights.
  • Stronger baselines, same harness (temperature 0, scorer unchanged):
ConditionOverallAbstainOver-refusal
WandaMind 7B72.61007.4
Qwen2.5-7B / +rule / +rule+3 shots41.9 / 40.3 / 38.70 / 33.3 / 66.73.7 / 11.1 / 11.1
Qwen2.5-14B / +rule41.9 / 41.90 / 50.03.7 / 7.4
Qwen3.6-27B / +rule / thinking51.6 / 43.5 / 38.766.7 / 83.3 / 011.1 / 14.8 / 11.1
Qwen3-32B / +rule / thinking41.9 / 48.4 / 45.250.0 / 83.3 / 16.77.4 / 7.4 / 7.4
Qwen2.5-72B / +rule48.4 / 38.750.0 / 83.37.4 / 22.2

No condition exceeds WandaMind (paired exact McNemar, Holm-corrected p<0.01). Only WandaMind reaches 100% abstention; prompted runs reaching 83.3% pay 7.4–22.2% over-refusal and lower Overall.

  • Evidence beyond QC-62. In the workbench (Q1), the untuned 7B's repeated identical calls had to be stopped in 47 turns vs 1 for WandaMind, and it made almost twice as many destructive-tool calls (416 vs 225).

W3. On human time vs machine time. A: You are right: in Table 2 the conventional time is human decision and tool-invocation time, CyberNeuro's is machine time (Appendix F). We claim no speedup; 99.94% measures removed manual work. We will revise Sec. 3.3 and Table 2 to rest on like-for-like quantities:

  • Interaction cost from recorded action traces: applications 3–4 → 1, switches 2–3 → 0, actions 14–20 → 1–2, 3/3 success in both. Times will be labelled human vs machine.
  • Machine-vs-machine time (RQ2): wall-clock from request to BIDS-ready output on the same ten cohorts, incl. model calls, tool runs, retries and stalls: 2,057 s (WandaMind) and 1,987 s (cloud) vs 2,565 s (NeuroClaw), i.e. 19.8% and 22.5% less, with 0 vs 24 interventions.

Q1. Untuned 7B inside the CyberNeuro system? A: Yes, at two levels.

  • Model level (QC-62). Untuned Qwen2.5-7B: 41.9% Overall on rerun (40.3% in the paper) vs 72.6% (McNemar p<0.001); abstention 0% vs 100%, false action 100% vs 0%, over-refusal 3.7% vs 7.4%; correct first tool on all 28 action items 50.0% vs 64.3%.
  • System level. We replaced the deployed WandaMind Q4_K_M with the official, hash-verified Qwen2.5-7B-Instruct Q4_K_M as Tier-0 (32K context) and sent the RQ2 DICOM-to-BIDS request to each of the ten cohorts five times (50 runs per model, fresh session, ≤ 20 min per turn, ≤ 2 fixed interventions). Upper bound: the correct pipeline call without a model produces BIDS on 10/10 cohorts.
WandaMind Q4Untuned 7B Q4
Submitted task ran and produced BIDS4/44/4
Submitted after loading the Skill4/81/4
Turns stopped by repeated-call guard147
Destructive-tool calls (at input data)225 (36)416 (77)

We can see that the WandaMind Q4 outperform the untuned model in this aspect. We hope these results address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.