- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper maps language instructions to electrical stimulation durations for xenobots using archived data. Claude scores archived trajectories against 33 prompts (18 for training, 15 held out); logistic fits produce duration targets, which a frozen SBERT encoder and two-head MLP learn to predict. Evaluation uses the majority label of the three nearest-duration trials. Reported accuracy on held-out prompts and trials is 80.0%, versus 66.7% for always predicting "slower."
Critical strengths
I like the problem and the use of real biological data. Checking the VLM against trajectory-derived labels is useful (Fig. 4, r = 0.843), as are the full-search permutation test, the 2×2 prompt × trial split, and the majority baseline. The supplement is commendably transparent: it includes the full VLM prompt, every prompt, per-prompt routing results, and a documented tracking bug.
Critical weaknesses
-
My main concern is the target construction. When the VLM-derived midpoint falls outside an interval where all training-archive labels are correct, it is clipped into that interval (L260–263; Supp. §10). For motion-increase prompts this happens almost always, and the shift is large: in Supp. Fig. 3, a midpoint of 105.2 s becomes 37.5 s. The targets that separate the model from the always-"slower" baseline are thus set by labels, not by the VLM.
-
I don't think the comparisons isolate the VLM's contribution. The numerical baseline uses one extreme trial per direction, changing several things at once. The VLM prompt also states each window's median step size and time interval (Supp. §3.1). This closely tracks the velocity used to define the labels, although L201–205 says the VLM is not given a precomputed velocity. A network trained on ground-truth labels matches the VLM-trained one in every cell (Supp. §5.1), but this comparison is missing from the main table.
-
The reported accuracy mainly measures prompt routing. The three GT2 errors are exactly the three held-out prompts that SBERT nearest-centroid routing sends to the wrong head. On the 120 paraphrases, ground-truth accuracy likewise equals routing accuracy (85.0%) (Supp. §5.1, §11). Beyond routing, the network only has to place each head in the right duration region; the supplement itself describes the capability this way (§12). The prompt split is also stated inconsistently: L238 calls all 33 prompts training prompts, whereas Supp. §4 lists 18 training and 15 held-out prompts.
-
The data and outcome definitions need work. The main text reports 101 classified trials (61 train / 40 test), but the supplement's analyses, which it says underlie the reported numbers, use 90 (54/36; Supp. §10.3 and the Fig. 3 caption). Fig. 5 marks p = 5.79×10⁻⁶, whereas L415 and Supp. §1.4 report 1.57×10⁻⁴ as beating all 200 shuffles; several shuffles in Fig. 5 are more extreme than that. Positional tracking error is not quantified (Supp. §2 documents a held-frame failure in 43–51% of batches), and no sham or unstimulated controls are reported.
Questions for authors
-
How many training prompts per bucket are clipped, and what is the accuracy without clipping? Table 1 puts CMA-ES near the majority baseline, but Supp. §10.2 reports one CMA-ES run on the same discrete reward reaching 80.0%. Could this label-free route be reported over 30 seeds?
-
How does a non-VLM score built from the same step-size summaries perform in the identical pipeline?
-
Which archive produced the reported numbers? Supp. Fig. 3 labels two prompts from the held-out list as training prompts; were held-out prompts' VLM scores used anywhere in training? How stable are the results across repeated splits?
-
Were the window search and the reliability floors (Supp. §1.1) restricted to training trials? How do the p-values in Fig. 5 and L415 relate? How large are "faster" velocity changes relative to tracking error? Were sham or unstimulated controls available?
For now, I see a retrospective proof of concept on xenobots; transfer to other cells or organoids remains untested. To raise my score, I'd need results without label-based clipping, consistent data reporting with a clean train/test boundary, and a gain over a matched numeric baseline.