- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces a method for verifying whether agents follow experimental protocols in cellular perturbation-response prediction tasks. The method checks for protocol violations (data leakage, metric drift, etc.) on expert written rules. The experimental setup is an LLM that proposes a workflow through which a predictive ridge regression model on the task is trained. Verifier is Python code checking JSON fields and logs.
The authors evaluate the method on two datasets and report that the method reduces experimental protocol violations compared to the baselines, along with providing ablation tests of different categories of verifiers and different models.
Critical strengths
The best portion of this paper are the baselines for protocol violation. Seeing how insensitive models are to those, and ablation tests where authors compared the models with and without VeriCell; later evaluating it on additional models to check that it produces similar results across a wider variety of models.
Critical weaknesses
-
The novelty of the contributions is limited. Most of the targeted violations, such as data leakage are familiar methodological problems with established best practices for handling them.
-
The application of the method is very limited in scope. Both in terms of the domain - cellular perturbation response prediction, and the experimental setting itself (in the provided code) - an LLM proposing a ridge regression workflow. The method’s applicability to other machine learning prediction tasks is unclear as currently presented and of limited use to a wider machine learning community. Its contribution might be more relevant in the cellular perturbation modeling community.
-
The supplementary code runs a ridge regression pipeline on synthetic data, with protocol checks that rely on JSON fields and audit logs generated by the pipeline itself. This does not support the stronger certification mechanisms described in the paper, such as independently monitoring actual data access.
-
The paper itself was very difficult to read, especially the technical sections. In terms of technical implementation I had to resort to code for understanding because the text was not very informative and I felt some key implementation details needed for clarity were skipped.
Questions for authors
-
Which predictive workflows were actually evaluated in the reported experiments, beyond the released ridge regression demo? Why does the provided code only include the partial implementation?
-
How does the verifier monitor data access independently of the workflow’s declared JSON fields and generated audit logs, and where is this implemented?
-
Could the reported reduction in violations be achieved by a simpler evaluation pipeline that enforces fixed splits, training-only preprocessing and similar simpler techniques? What additional guarantees or relative performance improvement does VeriCell’s certification provide to those?