acceptodds
Under review as a conference paper at ICLR 2027

DiMiRoS: A Digital Robotic Microscopic Dataset for Benchmarking High-Precision Surgery & Expert Dexterity Understanding

Abstract

For a century, what a microsurgeon saw was locked inside a pair of eyepieces. Digital exoscopic robotic microscopes open that lock: a camera and display make the visual feedback guiding every movement a signal that can be recorded and deliberately changed. This matters beyond surgery. Dexterity emerges as perception and action are calibrated through experience, yet recordings that could teach embodied AI expert manipulation remain scarce, scarcest where precision is extreme. Microsurgery is that extreme: bimanual, magnified, sub-millimetre, with almost no tolerance for error. We introduce DiMiRoS, the first surgical dataset recorded on a digital exoscopic robotic microscope: 308 end-to-end microvascular anastomoses by sixteen participants from novices to experts, captured from eleven synchronized views of the operative field, hands, body and operating room. Each participant operates in 2D and stereoscopic 3D with task and phantom unchanged, making visual feedback a controlled variable. Phase labels, object annotations, hand and body keypoints, and a blinded expert-derived evaluation ground benchmarks of suture quality and phase prediction. Reasoning VLMs overestimate quality, miss execution errors and confuse distinct actions performed with the same tools, while multi-view predictions do not reliably improve assessment. These failures expose a gap between seeing skilled movement and reasoning about its progress and consequences. DiMiRoS makes that gap measurable, opening a path towards embodied AI that learns how experts move and how vision shapes their movement.

Primary area:
datasets and benchmarks
Keywords:
microscopic surgery dataset, perception and reasoning benchmark, embodied AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fXsCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiMiRoS = dataset for microvascular anastomosis understanding + quality assesment. 308 trials,16 participants (3 exp levels), standardized phantom,2D and stereo 3D, 11 views (microscope/hands/body/OR). Comes with sparse manual boxes, auto seg + hand recon, phase labels, and blinded single-expert quality scores w/ error penalties.

Two few shot benchmarks:quality prediction (20 test trials) and phase recog (77 clips). Qwen3.5-9B / MedGemma get MAE 6.90/7.20, worse than few-shot mean (3.90); phase balanced acc 0.35 / 0.20. Then failure analysis (suture counting, errors, action confusion).

Critical strengths

Collection is built around a clear fine-manipulation problem,repeated trials same task/phantom, several exp levels, local + operator views. value is in the data itself, doesnt need new architecture.

Decomposed outcome-based scoring (count,spacing,bite,tension,apposition,errors) by an expert blinded to exp/display. Lets you separate “not finished” vs “finished but flawed”. single expert not consensus but still useful.

Critical weaknesses
  1. Multi-view sync + aux annotations not really validated. Abstract says “11 synchronized views” but App E.1 tells the model microscope and hand views aren’t time synced. so is raw data synced and just not used, or no alignment yet? no common timeline / offsets / sync error given. For a multi-view dataset this kinda matters, it decides which cross-view tasks are actually possible. Manual boxes only 1,341 frames over 21 videos, seg and hand geometry mostly auto (SAM 3,HaMeR). appendix describes an eval but no numbers to judge reliability on small objects/hands. fine as aux metadata, just maybe not frame it as validated dense seg/3D GT. (not saying theres leakage into the RGB benchmark)

  2. Hard to attribute low perf to “failed reasoning”. Labels come from opening the vessel after the procedure, model only sees the procedure. Reasonable task, but errors can’t directly mean the model missed visible errors. no expert-on-same-video reference, no analysis of what’s visible vs not. Also 1fps,640px width,memory/fusion through compressed text — could be the bottleneck. Phase test 77 clips(pull=4), not subject disjoint. So results read more like “these 2 models under this protocol” than general claims about generalization or multi-view.

Questions for authors

Q1. What does “synchronized” mean exactly — per-trial/per-camera mapping exists but unused? method,residual error,coverage per view? A.1 only mentions future EULA release — what’s available now/at release,how to access? restricted access is fine,just want to confirm.

Q2. Could an expert judge the same items from the same input? e.g blinded rating on a subset: raw video vs sampled model input vs post-op inspection,+ agreement. would help interpret model errors a lot.

Q3. In benchmark_trials.json the 3 test trials for 3D inexperienced are all subject 16,and example 3d_S16S5 too. not same trial,but that stratum isnt really 3 independent people. any subject-disjoint / LOSO results? maybe a simple video baseline or fps/resolution ablation? (I know shared participants are disclosed,not claiming leakage)

Q4. A.5 treats no-suture attempts as real zeros, but is_scored() in rubric.py drops all-zero records as maybe not performed/not scored. which is it? does it change the 308 count / score dist? raw sheet not in supp so just flagging.

Q5. Everyone did 2D then 3D,A.6 already says display and practice are confounded. any counterbalancing? if not maybe tone down 3D claims in abstract/main text to descriptive. not asking for a full study,just want the framing to match the design.

Reviewer_3qbkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiMiRoS is a microsurgical dataset recorded on a digital exoscopic robotic microscope. 16 participants performed 308 microvascular anastomosis trials on a vessel phantom in 2D and stereoscopic 3D, captured from eleven synchronized views. Annotations include sparse manual boxes, SAM 3 masks, HaMeR hand meshes, eight derived phases, and one expert’s suture-quality scorecard. Two benchmarks test few-shot reasoning VLMs: suture quality prediction (20 test trials) and phase prediction (77 clips). Both models do poorly; the authors conclude that VLMs miss faults and confuse actions performed with the same tools.

Critical strengths

A genuinely scarce resource: expert microsurgical recordings across three expertise levels, with 2D/3D feedback, multi-view capture, and expert quality scores with explicit fault penalties. Restricted-access release under an EULA, consent, and ethics approval are documented.

The finding that higher mean scores can coexist with more major errors is a useful reason to model errors separately.

Critical weaknesses

The benchmarks are too narrow to support the headline conclusions about VLMs and multi-view fusion. Only two small open VLMs are tested, with few-shot prompts and 20 test trials for quality; no frontier VLMs and no baselines trained on the other ~290 trials. The few-shot mean already beats both VLMs, so the results mostly show that few-shot prompting of small models fails. "No gain from fusion" rests on two views out of eleven, with differences within the reported standard deviations. Phase labels are derived from sparse annotations whose starts “may lag physical transitions” (Fig. 3), and explanations such as "confusing persistent tools with changing actions" are not tested.

The 2D-vs-3D conclusions are confounded. The protocol schedules ten 2D trials followed by ten 3D trials, so visual mode is not counterbalanced and is entangled with practice and fatigue. The reported 2D-to-3D score rises, and the reversal in major-error rates, come from 5–6 participants per cohort with no statistics or participant-level analysis.

Questions for authors

Was the 2D-to-3D order fixed for every participant? If so, can the authors separate mode from learning effects, and report participant-level statistics?

Could frontier VLMs and at least one supervised baseline trained on the training trials be added, with CIs over more than 20 test trials, so the claim “reasoning VLMs fail” is not specific to two small models?

The abstract and Sec. 1 state that the eleven views are synchronized, but Fig. 2 disclaims cross-camera temporal alignment, App. E.1 tells the model the two views are not time-synchronised, and App. A.2 describes no synchronization procedure. How were the views synchronized, and what is the alignment error? If they are not aligned, could the claim be revised, since it affects multi-view fusion results in Table 4?

Reviewer_JAjoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiMiRoS introduces a dataset of 308 phantom-based microvascular anastomosis trials performed by 16 participants across three expertise levels, using 2D and stereoscopic 3D visualization. The dataset combines recordings from eleven viewpoints with spatial annotations, annotation-derived surgical phases, reconstructed hand geometry, and expert quality assessments. Few-shot evaluations of two reasoning VLMs reveal substantial limitations in predicting surgical quality and identifying procedural phases.

Critical strengths
  • Valuable and challenging acquisition setting: Microsurgical manipulation provides a demanding setting for studying fine-grained dexterity. Capturing the operative field alongside hand and body views offers a potentially useful resource beyond conventional surgical video.
  • Rich performance supervision: Component-level quality scores and explicit fault counts support more informative assessment than a single expertise label or overall score.
  • The appendices provide sampling procedures, prompts, output-processing rules, and component-level analyses. The observation that plausible total scores can conceal incorrect procedural accounts is particularly useful.
Critical weaknesses
  • Some targets may not be observable from the benchmark inputs. Quality assessment includes inspecting the lumen after cutting open the repaired vessel. Certain internal defects may therefore be unavailable in the recorded operative and hand views. Without a video-only human reference, poor model performance cannot be attributed entirely to deficient visual reasoning. The single-assessor reference also leaves inter-rater reliability unmeasured.
  • Cross-view synchronization is described inconsistently. The abstract describes eleven synchronized views, whereas Appendix E.1 explicitly states that the microscope and hand views are not time-synchronized. This distinction matters for the dataset’s proposed multiview applications and for interpreting fusion failures.
  • The evaluated subsets and baseline coverage are limited. Quality prediction uses only 20 test trials, and phase recognition uses 77 clips from 11 videos. Two few-shot VLMs provide an initial diagnostic, but limited evidence about the broader difficulty or learnability of the benchmark. Supervised baselines, repeated inference, and uncertainty estimates are missing.
  • Visualization mode is confounded with practice. Participants always complete 2D trials before 3D trials. Although acknowledged in Appendix A.6, this prevents isolating visualization effects from accumulated practice or fatigue.

Overall, the dataset is potentially valuable, but target observability, synchronization consistency, and benchmark validation require stronger evidence before the central conclusions are sufficiently supported.

Questions for authors
  1. Which quality components can video-only expert assessors reliably predict, and how closely do their judgments agree with the post-dissection reference?
  2. Are the eleven views synchronized or independently timestamped? Please reconcile the abstract with Appendix E.1 and report synchronization accuracy.
  3. Can evaluation be expanded to participant-disjoint subsets, supervised video baselines, and repeated VLM runs?
  4. How much of the observed 2D–3D difference remains after accounting for trial order and participant-level learning?
1 comment

Dear Authors, Thank you for following up and for the additional experiments. The video-only reassessment partly addresses my concern about observability, particularly for suture count and spacing. I still share the concern raised in the discussion about internal faults. The expert reviewed the complete recordings, while the models received sampled frames, so it remains unclear which faults the models had enough visual evidence to identify. Agreement across assessors also remains unmeasured.

The 302-trial evaluation and supervised quality baseline address part of my concern about benchmark coverage. On the expanded set, Qwen improved on the mean baseline for six of seven components, although their total-score errors were nearly identical. I agree with the other reviewers that the conclusions should stay close to the settings tested. The supervised split includes participants seen during training, fusion helped MedGemma but increased Qwen’s error, and the phase results still come from 77 clips across 11 videos.

The shared start cue resolves my question about what synchronization means here. I also appreciate the mixed-effects analysis and the revised interpretation of the 2D–3D comparison. Because every participant completed 2D first, the higher scores in the later 3D block may partly reflect prior practice.

Overall, I see value in the recordings and the component-level annotations. My remaining reservations concern how reliably internal faults can be inferred from video and how well the results generalize to unseen participants. On balance, I now position this paper as a borderline article and if other reviewers think it passes the border, I can also go for acceptance but will not strongly support it.

Best

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.