- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CellAudit, a framework for checking whether agent-generated cellular-response models actually use inputs such as compound identity and dose. It separates three questions: whether the code allows an input to affect the computation, whether the trained predictions depend on it, and whether using the correct input reduces prediction error. The authors combine code inspection, input replacement, and error analysis, and explore returning these results to an agent to guide further model revisions. The main finding is a competitive BBBC047 predictor whose outputs remain exactly unchanged when compound identity is replaced. The authors identify an inactive attention pathway and further examine the finding through additional control experiments. The paper also audits a broader sample of candidate models and compares score-only feedback with audit-enriched feedback on sci-Plex. The latter produces better average results, but the five paired runs do not establish a reliable advantage.
Critical strengths
- The authors identify a model whose predictions remain exactly unchanged after compound replacement and locate an inactive attention pathway in its code. The finding persists when different control wells are used for model inputs and target references. The combination of code evidence and behavioral tests makes this specific finding convincing and demonstrates the need for such checks.
- The paper shows why input dependence and predictive benefit should be evaluated separately. Among 48 sampled candidate models, 47 change their predictions after compound replacement on both evaluation splits, but only 20 have target-loss benefit intervals entirely above zero on both splits. This does not prove that the remaining models gain nothing from the input. It does show that a change in predictions alone is not enough. This broader analysis extends the paper beyond the initial code defect and provides a useful lesson for model evaluation.
Critical weaknesses
- The methodological novelty is limited. Model search uses the existing CellScientist framework, while input replacement and prediction-error comparisons build on established methods. The main addition is to connect these methods with code inspection and use the resulting diagnostics to guide model revisions.
- The experiments do not establish a clear, reliable benefit from iterative audit feedback. First, the BBBC experiments change the model structure and training or selection rules together with the feedback. The improvements therefore cannot be attributed to audit feedback alone. The simple baselines also deserve attention: in Table 48, the residual MLP reaches 0.3154 PCC on BBBC047 Fold 5, compared with 0.3173 for the guided models, while showing larger compound-replacement effects and target-loss changes. This does not make the MLP better overall, but it leaves the added value of the complete workflow over simple residual modeling insufficiently established.
Questions for authors
- What information or capability does CellScientist provide that a conventional checking workflow would not readily offer?
- Are there existing comparisons that retain general diagnostics, such as grouped errors, while removing code checks or input-replacement results, with the model space, training settings, and candidate budget unchanged? Such evidence would affect the assessment of whether input-use auditing itself improves model design.
- Could the authors summarize the improvement after the first new candidate and compare it with the corresponding improvement under score-only feedback?