- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DiMiRoS = dataset for microvascular anastomosis understanding + quality assesment. 308 trials,16 participants (3 exp levels), standardized phantom,2D and stereo 3D, 11 views (microscope/hands/body/OR). Comes with sparse manual boxes, auto seg + hand recon, phase labels, and blinded single-expert quality scores w/ error penalties.
Two few shot benchmarks:quality prediction (20 test trials) and phase recog (77 clips). Qwen3.5-9B / MedGemma get MAE 6.90/7.20, worse than few-shot mean (3.90); phase balanced acc 0.35 / 0.20. Then failure analysis (suture counting, errors, action confusion).
Critical strengths
Collection is built around a clear fine-manipulation problem,repeated trials same task/phantom, several exp levels, local + operator views. value is in the data itself, doesnt need new architecture.
Decomposed outcome-based scoring (count,spacing,bite,tension,apposition,errors) by an expert blinded to exp/display. Lets you separate “not finished” vs “finished but flawed”. single expert not consensus but still useful.
Critical weaknesses
-
Multi-view sync + aux annotations not really validated. Abstract says “11 synchronized views” but App E.1 tells the model microscope and hand views aren’t time synced. so is raw data synced and just not used, or no alignment yet? no common timeline / offsets / sync error given. For a multi-view dataset this kinda matters, it decides which cross-view tasks are actually possible. Manual boxes only 1,341 frames over 21 videos, seg and hand geometry mostly auto (SAM 3,HaMeR). appendix describes an eval but no numbers to judge reliability on small objects/hands. fine as aux metadata, just maybe not frame it as validated dense seg/3D GT. (not saying theres leakage into the RGB benchmark)
-
Hard to attribute low perf to “failed reasoning”. Labels come from opening the vessel after the procedure, model only sees the procedure. Reasonable task, but errors can’t directly mean the model missed visible errors. no expert-on-same-video reference, no analysis of what’s visible vs not. Also 1fps,640px width,memory/fusion through compressed text — could be the bottleneck. Phase test 77 clips(pull=4), not subject disjoint. So results read more like “these 2 models under this protocol” than general claims about generalization or multi-view.
Questions for authors
Q1. What does “synchronized” mean exactly — per-trial/per-camera mapping exists but unused? method,residual error,coverage per view? A.1 only mentions future EULA release — what’s available now/at release,how to access? restricted access is fine,just want to confirm.
Q2. Could an expert judge the same items from the same input? e.g blinded rating on a subset: raw video vs sampled model input vs post-op inspection,+ agreement. would help interpret model errors a lot.
Q3. In benchmark_trials.json the 3 test trials for 3D inexperienced are all subject 16,and example 3d_S16S5 too. not same trial,but that stratum isnt really 3 independent people. any subject-disjoint / LOSO results? maybe a simple video baseline or fps/resolution ablation? (I know shared participants are disclosed,not claiming leakage)
Q4. A.5 treats no-suture attempts as real zeros, but is_scored() in rubric.py drops all-zero records as maybe not performed/not scored. which is it? does it change the 308 count / score dist? raw sheet not in supp so just flagging.
Q5. Everyone did 2D then 3D,A.6 already says display and practice are confounded. any counterbalancing? if not maybe tone down 3D claims in abstract/main text to descriptive. not asking for a full study,just want the framing to match the design.
Dear Authors, Thank you for following up and for the additional experiments. The video-only reassessment partly addresses my concern about observability, particularly for suture count and spacing. I still share the concern raised in the discussion about internal faults. The expert reviewed the complete recordings, while the models received sampled frames, so it remains unclear which faults the models had enough visual evidence to identify. Agreement across assessors also remains unmeasured.
The 302-trial evaluation and supervised quality baseline address part of my concern about benchmark coverage. On the expanded set, Qwen improved on the mean baseline for six of seven components, although their total-score errors were nearly identical. I agree with the other reviewers that the conclusions should stay close to the settings tested. The supervised split includes participants seen during training, fusion helped MedGemma but increased Qwen’s error, and the phase results still come from 77 clips across 11 videos.
The shared start cue resolves my question about what synchronization means here. I also appreciate the mixed-effects analysis and the revised interpretation of the 2D–3D comparison. Because every participant completed 2D first, the higher scores in the later 3D block may partly reflect prior practice.
Overall, I see value in the recordings and the component-level annotations. My remaining reservations concern how reliably internal faults can be inferred from video and how well the results generalize to unseen participants. On balance, I now position this paper as a borderline article and if other reviewers think it passes the border, I can also go for acceptance but will not strongly support it.
Best