acceptodds
Under review as a conference paper at ICLR 2027

Discover, Falsify, Revise: Auditing Input-Use Claims from Source Code to Predictive Contribution in Agent-Discovered Cell Models

Abstract

AI virtual cells aim to predict cellular responses to specified interventions, yet held-out predictive performance alone does not establish use of the supplied perturbation information. This prediction–claim gap is especially consequential in agentic model discovery, where language-model agents generate and revise predictors using predominantly score-based feedback. We introduce CellAudit, which audits registered input-use claims through three distinct questions: whether the input can enter the cited computation, whether fitted predictions depend on it, and whether that dependence improves prediction of observed responses. On a paired morphology–transcriptomics perturbation benchmark (BBBC047), an agent-selected predictor achieves a mean held-out Global Pearson correlation coefficient (PCC) of 0.3153 yet remains exactly invariant to compound replacement; a control-profile-only predictor reaches 0.3142. Source inspection identifies a compound-query pathway blocked by singleton key–value attention; the invariance persists after refitting with physically disjoint control wells for inputs and target references. In a stratified audit of 48 generated candidates across two linked tasks, 47 change predictions under compound replacement on both held-out folds, but only 20 show target-loss gains with intervals above zero on both folds. On BBBC047, falsification-guided revisions recover positive mean compound contributions while retaining predictive gains over the control-profile-only baseline. In matched sci-Plex searches, audit-enriched feedback shows higher mean held-out predictive performance and larger mean compound and dose contributions across five paired trajectories, although paired intervals span zero. Refitting fixed designs on an independently acquired cohort further shows predictive generalization need not imply generalization of input-use claims: dose contribution persists, whereas support for compound identity does not persist. CellAudit therefore adds a falsification layer to agentic model discovery, moving from generate–score–revise toward discover–falsify–revise.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AI for Science, agentic model discovery, cellular perturbation prediction, input-use auditing, model falsification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XfpEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CellAudit, a framework for checking whether agent-generated cellular-response models actually use inputs such as compound identity and dose. It separates three questions: whether the code allows an input to affect the computation, whether the trained predictions depend on it, and whether using the correct input reduces prediction error. The authors combine code inspection, input replacement, and error analysis, and explore returning these results to an agent to guide further model revisions. The main finding is a competitive BBBC047 predictor whose outputs remain exactly unchanged when compound identity is replaced. The authors identify an inactive attention pathway and further examine the finding through additional control experiments. The paper also audits a broader sample of candidate models and compares score-only feedback with audit-enriched feedback on sci-Plex. The latter produces better average results, but the five paired runs do not establish a reliable advantage.

Critical strengths
  1. The authors identify a model whose predictions remain exactly unchanged after compound replacement and locate an inactive attention pathway in its code. The finding persists when different control wells are used for model inputs and target references. The combination of code evidence and behavioral tests makes this specific finding convincing and demonstrates the need for such checks.
  2. The paper shows why input dependence and predictive benefit should be evaluated separately. Among 48 sampled candidate models, 47 change their predictions after compound replacement on both evaluation splits, but only 20 have target-loss benefit intervals entirely above zero on both splits. This does not prove that the remaining models gain nothing from the input. It does show that a change in predictions alone is not enough. This broader analysis extends the paper beyond the initial code defect and provides a useful lesson for model evaluation.
Critical weaknesses
  1. The methodological novelty is limited. Model search uses the existing CellScientist framework, while input replacement and prediction-error comparisons build on established methods. The main addition is to connect these methods with code inspection and use the resulting diagnostics to guide model revisions.
  2. The experiments do not establish a clear, reliable benefit from iterative audit feedback. First, the BBBC experiments change the model structure and training or selection rules together with the feedback. The improvements therefore cannot be attributed to audit feedback alone. The simple baselines also deserve attention: in Table 48, the residual MLP reaches 0.3154 PCC on BBBC047 Fold 5, compared with 0.3173 for the guided models, while showing larger compound-replacement effects and target-loss changes. This does not make the MLP better overall, but it leaves the added value of the complete workflow over simple residual modeling insufficiently established.
Questions for authors
  1. What information or capability does CellScientist provide that a conventional checking workflow would not readily offer?
  2. Are there existing comparisons that retain general diagnostics, such as grouped errors, while removing code checks or input-replacement results, with the model space, training settings, and candidate budget unchanged? Such evidence would affect the assessment of whether input-use auditing itself improves model design.
  3. Could the authors summarize the improvement after the first new candidate and compare it with the corresponding improvement under score-only feedback?
Reviewer_QGFkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether agent-generated cell-response models truly use the perturbation information they receive, since a strong prediction score alone cannot show that they do. It introduces CellAudit, a framework that examines model source code, measures how predictions change when inputs are substituted, and checks whether the correct inputs improve predictions. The authors evaluate CellAudit on chemical and genetic perturbation datasets, then use the audit results to guide further model revisions.

Critical strengths

None that strongly supports acceptance in my assessment.

Critical weaknesses

This paper is not even wrong.

  • The significance and new contribution are unclear. The author built an agent with problems and proposes a method to fix them. The main failures stem from models generated within the study’s own agent-search setup. It is unclear whether these findings reveal a general problem in perturbation modeling or mainly problems in this generation pipeline. Established drug-perturbation models, such as CellFlow [1] or CPA [2], don’t seem to have similar issues (if they do, please correct me). The message that prediction accuracy does not prove useful input dependence is basic and is the direct result of using vibe-coding-style autoresearch. Beyond that, I don’t see a clear takeaway for building better agentic AI or biology models.
  • The experiments do not clearly show the value of the audit. In BBBC, both model structure and feedback change, so it's unclear what drove the improvement. The better-controlled sci-Plex study has only five paired trajectories. These experiments do not provide strong evidence that the audit improves model discovery. I would suggest using the same benchmark, for example, from CellForge [3] or the Virtual Cell Challenge, to audit your models.

[1] Klein, D., et al. CellFlow enables generative single-cell phenotype modeling with flow matching. bioRxiv (2025).

[2] Lotfollahi, M., et al. Predicting cellular responses to complex perturbations in high-throughput screens. Molecular Systems Biology 19, e11517 (2023).

[3] Tang, X., et al. AI Scientists for Building Virtual-Cell Models (CellForge). arXiv, v3 (2026).

Questions for authors
  • What general insights does CellAudit provide beyond code checks and input-replacement tests? What is the author's takeaway from this paper?
  • Was Fold 5 kept untouched during the development of the CellAudit method, prompts, and selection rules?
  • How do you audit a drug encoder for responses to single-gene edits?
Reviewer_Np4pOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Disclaimer: I am not a cell biologist, AC has been notified, my review will be associated with a low confidence score. My presentation and informativeness scores are also affected by the same issue. I tried to provide an honest reading and interpretation of this work to the best of my capabilities.

Critical strengths

//

Critical weaknesses

I don’t know if it’s because I don’t have any experience in this field but lots of claims and statements are undefined in this work. For example, what is an “input-use claim”? I promised that I tried to search online and did some literature search and I failed to find any reasonable match for this definition. In the paper, it is introduced as an obvious statement.

As an additional example of where I think paper writing is not really well done, consider the Problem Setup Section (also why is it a micro section?). For example, “biological context” occurs only one time in the entire paper. Even the initial sentence starts with “Each example” but I am not sure of what these are examples of. The same paragraph ends with a similar problem: I am not sure what “Folds” are. It feels like some portions of this paper are disconnected.

This is also a problem in later sections:

Section 4.1 seems to switch topics to CellScientist, but then I hardly see again how 4.1 connects with 4.2. Then I arrive at half Section 4.2 where I read, completely out of context: “With one key–value pair, attention has constant normalized weight and cannot transmit compound information through its query (Vaswani et al., 2017).”, is this an example? I really could not understand what the authors wanted to say, this sentence seems completely out of place (but it also appears in Figure 1 and in the Introduction, so…). Page 4 ends with a paragraph that starts with “The sci-Plex comparison” and I can’t understand, again, why this is disconnected from the previous paragraph.

Other problems include, defining f_s in page 3 and never use f again in page 4 with a different definition plus a set of symbols that was never defined before (also f is used in two different ways in equation 1?).

Assuming this paper is correct - which I struggle to verify - it’s most likely not well written.

Questions for authors

//

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.