acceptodds
Withdrawn from ICLR 2027

Protocol-Certified Workflow Search for Cellular Perturbation-Response Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language model agents now automate the whole modelling pipeline for cellular perturbation response, selecting among candidate workflows by a single validation score. We argue that this score is the wrong contract, because the protocol violations that matter most in this domain all *raise* it: leaking held out rows into selection, fitting normalization on the full corpus, admitting post treatment measurements as inputs, exploiting plate identity under a plate based split, selecting responsive genes with the model's own output, or requerying validation until the last edit looks good. None of these raises an exception, and validity is a property of the dependence structure among artifacts rather than of any single statement: execution feedback observes only crashes, and a local assertion observes only one statement, so neither reaches them. A search rewarded by that score therefore has a structural incentive to find them, and our audit confirms it: across nine violation categories the worst offender is not a weak agent but our own closed loop scaffold with certification disabled, at a 0.278 macro average violation rate against 0.169 for a generic AutoML pipeline. VᴇʀɪCᴇʟʟ moves admissibility out of the generator into a verifier. A taint labelled provenance graph collapses a heterogeneous family of protocol errors into one non interference property, discharged by a sound type and effect analysis composed with fail closed reference monitors; an audited reusable holdout bounds the adaptive generalization gap; and a claim checker ties every reported number to certified evidence. Certification cuts the macro average violation rate to 0.007 at verifier precision and recall above 0.93, and the validation to test gap from 0.120 to 0.016, while improving accuracy where signal exists (LINCS L1000 PCC 0.463 → 0.557, BBBC021 PCC 0.264 → 0.355) and costing nothing where it does not. Throughout, *certified* means machine checked protocol validity relative to a written contract and a bounded validation reuse gap, not biological correctness.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
LLM Agents, Data Leakage, Perturbation Response Prediction, Evaluation Integrity, Adaptive Data Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_27AyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a method for verifying whether agents follow experimental protocols in cellular perturbation-response prediction tasks. The method checks for protocol violations (data leakage, metric drift, etc.) on expert written rules. The experimental setup is an LLM that proposes a workflow through which a predictive ridge regression model on the task is trained. Verifier is Python code checking JSON fields and logs.

The authors evaluate the method on two datasets and report that the method reduces experimental protocol violations compared to the baselines, along with providing ablation tests of different categories of verifiers and different models.

Critical strengths

The best portion of this paper are the baselines for protocol violation. Seeing how insensitive models are to those, and ablation tests where authors compared the models with and without VeriCell; later evaluating it on additional models to check that it produces similar results across a wider variety of models.

Critical weaknesses
  1. The novelty of the contributions is limited. Most of the targeted violations, such as data leakage are familiar methodological problems with established best practices for handling them.

  2. The application of the method is very limited in scope. Both in terms of the domain - cellular perturbation response prediction, and the experimental setting itself (in the provided code) - an LLM proposing a ridge regression workflow. The method’s applicability to other machine learning prediction tasks is unclear as currently presented and of limited use to a wider machine learning community. Its contribution might be more relevant in the cellular perturbation modeling community.

  3. The supplementary code runs a ridge regression pipeline on synthetic data, with protocol checks that rely on JSON fields and audit logs generated by the pipeline itself. This does not support the stronger certification mechanisms described in the paper, such as independently monitoring actual data access.

  4. The paper itself was very difficult to read, especially the technical sections. In terms of technical implementation I had to resort to code for understanding because the text was not very informative and I felt some key implementation details needed for clarity were skipped.

Questions for authors
  1. Which predictive workflows were actually evaluated in the reported experiments, beyond the released ridge regression demo? Why does the provided code only include the partial implementation?

  2. How does the verifier monitor data access independently of the workflow’s declared JSON fields and generated audit logs, and where is this implemented?

  3. Could the reported reduction in violations be achieved by a simpler evaluation pipeline that enforces fixed splits, training-only preprocessing and similar simpler techniques? What additional guarantees or relative performance improvement does VeriCell’s certification provide to those?

Reviewer_8WP7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VeriCell, a framework for enforcing experimental protocols during agent-generated workflow search for cellular perturbation-response prediction. It combines provenance-based information-flow analysis, runtime monitoring, an audited reusable holdout, and claim–evidence checking. Candidates enter model selection only after passing checks against an expert-written protocol contract, while violations produce structured repair feedback.

Critical strengths
  • Practical relevance: Agent-generated modelling workflows can obtain inflated validation scores through leakage, shortcuts, or adaptive overfitting without producing runtime errors. Addressing these failures is relevant to the reliability of automated scientific modelling.

  • Controlled evidence of effectiveness: Table 3 holds the backbone, scaffold, and search budget fixed while disabling certification. The macro-average violation rate increases from approximately 0.7% to 27.8%, supporting the effectiveness of certification in reducing violations under the reported audit.

Critical weaknesses
  • Limited demonstrated novelty: The contribution appears to lie mainly in formalizing domain-specific rules and integrating established verification techniques. Although this integration may be useful, the paper does not clearly distinguish the adaptations needed for cellular perturbation workflows from capabilities already available in existing methods, or explain the technical difficulties those adaptations resolve.
Questions for authors
  1. Which aspects of VeriCell constitute substantive adaptations beyond applying existing verification methods to this domain? A concrete example explaining the technical obstacle and how the proposed mechanism resolves it would help clarify the contribution.

  2. Could the authors compare VeriCell against a simpler independent protocol checker?

Reviewer_rTNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that validation score is the wrong selection signal for LLM agents that write modeling pipelines for perturbation-response prediction, because every protocol violation worth worrying about (split leakage, train-fit normalization on the full corpus, plate-identity shortcuts, responsive-gene circularity, adaptive reuse of validation) raises that score. VERICELL moves admissibility out of the generator into a verifier: a domain expert writes a protocol contract as typed invariants, execution is instrumented as a taint-labelled provenance graph, and most violations reduce to one non-interference check discharged statically plus runtime monitors. A reusable holdout bounds adaptive overfitting and a claim checker ties reported numbers to monitored nodes. On Cell Painting and LINCS L1000, certification drops the macro violation rate from 0.278 (their own scaffold with certification off) to 0.007 and the val–test gap from 0.120 to 0.016, without hurting accuracy.

Critical strengths
  1. The reduction of a heterogeneous family of protocol errors to a single non-interference property over a provenance graph is a clean idea and, as far as I know, new for this domain. It explains why per-pattern detectors keep missing things.
  2. The no-guard row in Table 3 is the right ablation: same backbone, scaffold, and budget, so the drop is attributable to the verifier and not to a stronger agent. The paper is also unusually honest about negative results (bolding the baseline in Table 6, reporting the failed single-cell benchmarks).
Critical weaknesses
  1. The accuracy story is under-supported. Tables 1 and 2 compare VERICELL against fixed models and AutoML, but the fair accuracy comparison is VERICELL vs. its own no-guard scaffold, which only appears in Table 8 on one regime (0.527 → 0.551). As written, I can’t tell how much of the BBBC021/LINCS gain is the LLM agent writing better code vs. certification. That matters because “certification costs nothing” is a headline claim.

  2. Half the audit cells are open-ended tasks scored by two annotators against a rubric, and the “injected” half plants an error into a reference pipeline and checks whether the agent’s pipeline reproduces it — it isn’t clear why an agent that starts from scratch would reproduce a planted error at all. The 0.278 number is central to the paper and its construction needs a much clearer description than Appendix A.3 gives.

Questions for authors

-What is test PCC for the no-guard scaffold on every cell of Tables 1 and 2?

-In the injected suite, does the agent see the reference pipeline? If not, what does “exhibits the planted error” mean operationally?

-Theorem 1 fixes c = 1 “from the composition steps.” Can you show that derivation? Thresholdout constants are usually not 1.

-Contracts took 40–70 minutes each but all four instantiate the same 13 templates. What does writing a contract for a new assay type (e.g., Perturb-seq) look like?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.