- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper attempts to assess whether requiring agents to commit to an analysis without seeing results helps (Commit Before you Inspect). The paper sets up a series of small dataset simulations and calculates the "false release rate" with our without such precommitment. Section 4 has a negative pilot and Section 5 uses a simple dataset with four covariates and 200 observations and four experiments (E1-E4). In E1, they report a reduction in the false release rate. Main discretionary design choices are regression adjustment and weighting. Section 5.8 also has a soccer-hurricane dataset showing a similar effect.
Critical strengths
Understanding how to have agentic workflows abide b good scientific practices is important. Preregistration of hypotheses and research design has been central to scientific integrity, so it is a fine research question to understand how similar commitment devices might be used for agents.
Critical weaknesses
The paper is exceedingly hard to follow. Experiments are not well described. There are lots of acronyms that are never properly introduced (e.g., E1 appears in the fourth sentence of the abstract, but the reader has no sense of what the experiment entails, PRIMARY_POSITIVE_BELOW_MEANINGFUL_EFFECT).
The actual experiments E1-E4 feel very artificial and contrived. Only the appendix really provides details on the data generating process, so the reader is left guessing in the main manuscript what the experiments really entail.
The results appear to be driven by agents being able to see the results and screening on statistical significance. Both the false release and true detection rate go up.
It is confusing what evidence a blind agent is supposed to make a decision on. Why should it be able to make an informed decision about regression specification based on what it is prompted with? Here all the details are in the appendix, and the setup needs to be made much clearer.
Questions for authors
Can you re-write the paper to try to communicate more clearly what the setup is in the body?