acceptodds
Under review as a conference paper at ICLR 2027

Toward Convincing Biology with Language: Offline Learning of Prompt-Conditioned Interventions for Cells, Organoids, and Biobots

Abstract

Artificial intelligence increasingly serves as a natural-language interface to complex technical systems, letting people accomplish sophisticated tasks by describing what they want rather than specifying how to do it. Extending this interface to living systems is harder: unlike code or images, a biological intervention has no closed-form linguistic meaning, and the paired language-intervention-outcome data needed to learn such a mapping is expensive to collect, since each example requires its own wet-lab experiment. One way around this is to treat an existing archive of interventions and their already-observed outcomes as a fixed, offline dataset, and use a vision-language model to judge, without any new experiments, whether an archived outcome matches a natural-language description. But whether that judgment is reliable enough to train a language-to-intervention mapping on – without new experiments and without human validation – has remained unclear. Here we show that a natural-language interface for a living organism – a xenobot, a synthetic multicellular construct with no nervous system – can be learned entirely offline this way, using a vision-language model's own judgment as the sole training reward: an instruction is mapped to the intervention already on record as producing the described behavior. This mapping generalizes to entirely new instructions, evaluated against archive data withheld from training ($80.0% held-out accuracy vs.a 66.7% chance baseline, matching a network trained directly on ground-truth labels). These results indicate that offline archives, combined with automated judgment, can substitute for costly new experiments when learning to ground natural language in a living system's own recorded behavior - an intermediate step toward directing living matter with the same ease as code, ahead of testing whether these predicted interventions reproduce their intended behavior on live, newly-stimulated cells and, ultimately, convincing biology with language alone.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
organoids, foundation models, vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_73weOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper maps language instructions to electrical stimulation durations for xenobots using archived data. Claude scores archived trajectories against 33 prompts (18 for training, 15 held out); logistic fits produce duration targets, which a frozen SBERT encoder and two-head MLP learn to predict. Evaluation uses the majority label of the three nearest-duration trials. Reported accuracy on held-out prompts and trials is 80.0%, versus 66.7% for always predicting "slower."

Critical strengths

I like the problem and the use of real biological data. Checking the VLM against trajectory-derived labels is useful (Fig. 4, r = 0.843), as are the full-search permutation test, the 2×2 prompt × trial split, and the majority baseline. The supplement is commendably transparent: it includes the full VLM prompt, every prompt, per-prompt routing results, and a documented tracking bug.

Critical weaknesses
  1. My main concern is the target construction. When the VLM-derived midpoint falls outside an interval where all training-archive labels are correct, it is clipped into that interval (L260–263; Supp. §10). For motion-increase prompts this happens almost always, and the shift is large: in Supp. Fig. 3, a midpoint of 105.2 s becomes 37.5 s. The targets that separate the model from the always-"slower" baseline are thus set by labels, not by the VLM.

  2. I don't think the comparisons isolate the VLM's contribution. The numerical baseline uses one extreme trial per direction, changing several things at once. The VLM prompt also states each window's median step size and time interval (Supp. §3.1). This closely tracks the velocity used to define the labels, although L201–205 says the VLM is not given a precomputed velocity. A network trained on ground-truth labels matches the VLM-trained one in every cell (Supp. §5.1), but this comparison is missing from the main table.

  3. The reported accuracy mainly measures prompt routing. The three GT2 errors are exactly the three held-out prompts that SBERT nearest-centroid routing sends to the wrong head. On the 120 paraphrases, ground-truth accuracy likewise equals routing accuracy (85.0%) (Supp. §5.1, §11). Beyond routing, the network only has to place each head in the right duration region; the supplement itself describes the capability this way (§12). The prompt split is also stated inconsistently: L238 calls all 33 prompts training prompts, whereas Supp. §4 lists 18 training and 15 held-out prompts.

  4. The data and outcome definitions need work. The main text reports 101 classified trials (61 train / 40 test), but the supplement's analyses, which it says underlie the reported numbers, use 90 (54/36; Supp. §10.3 and the Fig. 3 caption). Fig. 5 marks p = 5.79×10⁻⁶, whereas L415 and Supp. §1.4 report 1.57×10⁻⁴ as beating all 200 shuffles; several shuffles in Fig. 5 are more extreme than that. Positional tracking error is not quantified (Supp. §2 documents a held-frame failure in 43–51% of batches), and no sham or unstimulated controls are reported.

Questions for authors
  1. How many training prompts per bucket are clipped, and what is the accuracy without clipping? Table 1 puts CMA-ES near the majority baseline, but Supp. §10.2 reports one CMA-ES run on the same discrete reward reaching 80.0%. Could this label-free route be reported over 30 seeds?

  2. How does a non-VLM score built from the same step-size summaries perform in the identical pipeline?

  3. Which archive produced the reported numbers? Supp. Fig. 3 labels two prompts from the held-out list as training prompts; were held-out prompts' VLM scores used anywhere in training? How stable are the results across repeated splits?

  4. Were the window search and the reliability floors (Supp. §1.1) restricted to training trials? How do the p-values in Fig. 5 and L415 relate? How large are "faster" velocity changes relative to tracking error? Were sham or unstimulated controls available?

For now, I see a retrospective proof of concept on xenobots; transfer to other cells or organoids remains untested. To raise my score, I'd need results without label-based clipping, consistent data reporting with a clean train/test boundary, and a gain over a matched numeric baseline.

Reviewer_EgYSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to learn a “natural-language interface” for xenobots. A user types an instruction such as “make the bot slow down,” and the model predicts how long an electrical stimulus should last. Training needs no new lab experiments (which is a plus point). The authors take an existing archive of recorded stimulations, have a vision-language model (VLM) score how well each recorded trajectory matches each of 33 training prompts, and turn those scores into training targets for a small network on top of frozen sentence embeddings. On held-out prompts and held-out archive data, the model reaches 80.0% accuracy against a 66.7% chance baseline.

The authors also mention two properties that have limited the advancement. First, collecting data is expensive, and second, each data point is a wet-lab experiment.

Critical strengths

The motivation is interesting. Treating an existing experimental archive as an offline dataset, with a VLM as the judge, is a reasonable way around expensive wet-lab data collection. Specifically, it works for the two limitations pointed out by the authors early in the paper. The authors also check the VLM judge against independently tracked behavior labels instead of trusting it blindly. They also run a permutation test on their window-selection grid search. Both are good practice.

Critical weaknesses

The task is much simpler than the framing suggests. Only two behaviors (faster or slower) and a single number (duration) are involved. The model mostly has to decide whether a prompt means “faster” or “slower” and then output a duration on the correct side of the archive. The output head is even chosen by nearest-centroid similarity of sentence embeddings, which is essentially a sentiment-style classifier. A trivial baseline is needed to strengthen the claims and the experimental setup.

Ground-truth labels may leak into the “VLM-only” reward. The training target is clipped into an interval “where every point is correctly labeled,” so the true behavior labels are used to build the target. This undercuts the central claim that the VLM’s judgment is the sole training signal. How often does this clipping happen, and what are the results without it?

The statistics are thin. The test archive has 40 batches and the train set has only 16 “faster” examples. A result of 80.0 ± 0.0% across 30 seeds suggests the outcome is decided by a few discrete cases rather than measured precisely. Confidence intervals or tests against the chance baseline are needed.

The multi-task motivation is overstated. The paper cites gradient-interference work at length, but with two scalar heads selected by embedding similarity there is little real interference to solve.

The scope is overclaimed. The title mentions cells, organoids, and biobots, but every experiment is on xenobots with one stimulation parameter. No predicted intervention is tested on a live organism.

Questions for authors
  1. How does the trivial “classify direction + class-median duration” baseline perform?
  2. What are the results without label-based clipping of the target?
  3. Which VLM version and query template were used? Is the judge’s scoring stable across repeated queries?
Reviewer_zmvEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates learning a natural-language interface for biological interventions from existing experimental records. I find the problem important, distinctive, and well motivated, particularly given the cost of collecting new biological experiments. The proposed pipeline uses a vision-language model to score how well recorded xenobot trajectories match language instructions, converts these scores into target electrical-stimulation durations, and trains a prompt-to-duration network. Its main contribution is applying this offline learning approach to real xenobot recordings. Validation includes comparisons with trajectory-derived behavior labels, alternative optimization methods, and a numerical-target baseline.

Critical strengths
  1. The paper studies an important and unusual problem: connecting natural-language goals to biological interventions. I find the motivation convincing, and this direction has potential to bring AI into closer interaction with scientific experiments and the physical world.

  2. Reusing existing xenobot recordings is a practical choice given the cost of biological experiments. It allows the authors to investigate language-to-intervention learning using real biological data without collecting new experiments during training.

  3. The authors compare VLM judgments with trajectory-derived behavioral labels. This is a useful check of whether the scoring model captures measured behavioral changes, rather than relying solely on its own assessments.

Critical weaknesses
  1. The additional capability provided by the learned mapping remains unclear. Supplementary Section 12 describes the demonstrated behavior as largely category-level routing followed by a learned duration. A direct comparison with a language classifier followed by a representative duration for each category would help establish the value of the proposed pipeline.

  2. The supplementary analyses are useful, but the experimental scope remains narrow relative to the broader motivation. The main evaluation covers one biological system, one intervention parameter, and two behavioral directions. Additional paraphrases test language variation, while the effectiveness of the predicted interventions on newly stimulated xenobots remains untested.

  3. Several figures, particularly Figures 1, 3, and 5 in the main paper, contain labels and annotations that are too small to read comfortably. This makes important methodological and statistical details difficult to inspect.

  4. The numerical-target comparison in Section 4.3 does not isolate the contribution of VLM scoring. The numerical baseline selects a single extreme trial, whereas the proposed method fits scores across trials and adjusts the target using behavior labels. Despite the additional baselines in the supplement, a comparison that holds target construction fixed is still needed to attribute the improvement to the VLM.

Questions for authors
  1. Could the authors clarify the relationship between the 101 trials (61/40 split) in Section 3.2 and the 90 trials (54/36 split) in Supplementary Section 10.3? These may reflect different subsets or preprocessing stages. Which dataset and split were used for the main results?

  2. Supplementary Section 10 describes clipping VLM-derived targets into intervals defined by measured behavior labels. How should this use of labels be reconciled with the claim that the VLM’s judgment is the sole training reward? Could the reported accuracy depend substantially on this label-based adjustment?

  3. See Weakness 1.

  4. See Weakness 4.

Reviewer_CeFgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes to leverage a VLM as a judge for whether an archived intervention outcome matches a natural-language description. Specifically, the authors collect a dataset of xenobots (a synthetic multicellular organism) and train a small neural network to predict intervention durations. The experiments report results better than models trained on ground-truth labels.

Critical strengths

The idea of using VLMs to generate language–intervention labels is interesting and potentially practical.

Critical weaknesses
  1. This paper is poorly presented, with AI-generated figures. It is extremely hard for the reader to figure out the goal task of the paper, its difficulty, and what insight the paper provides. For example, the paper calls the proposed pipeline an “archive-based learning paradigm” and the judgment outcome a “reward.” However, the actual approach appears to be a straightforward VLM-based pseudo-labeling pipeline followed by supervised regression. It is unclear what methodological novelty this framework offers beyond combining existing components, including VLMs and regression. The paper should describe the method more precisely and justify why this relatively simple task requires such a “learning framework.”
  2. The motivation for using REINFORCE and CMA-ES as baselines is unclear. Since all VLM scores are precomputed from a fixed archive, the problem is simply a pseudo-label generation followed by supervised regression. The authors claim that backprop provides better performance than REINFORCE. However, this comparison does not establish methodological novelty or demonstrate the necessity of the proposed pipeline. It also makes the claim of “matching a network trained directly on ground-truth labels” confusing.
  3. The paper does not provide convincing evidence that the constructed dataset is sufficiently reliable to support the claims. The retained 101 tracked experiments have class imbalance (26 faster and 75 slower). It is unclear how difficult the task is; for example, how long it will take a human expert to conduct an examination. Also, Section 3.4 seems to use all 101 experiments to conduct training rewards, including the 40 experiments described as held out in Section 3.5. It raises concerns about evaluation leakage.
Questions for authors
  1. What is the actual task of the paper, clearly defined?
  2. How much acceleration can the automated task enable? What is the accuracy compared with human experts?
  3. Why is the proposed pipeline a “learning paradigm?” I don’t think using the pseudo-labels generated by VLMs to train a classifier is essentially a learning method. What is the methodological novelty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.