- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper randomly varies goal wording and the number of evaluations before repair on small-scale coding tasks, and compares treatment effects early in the trajectory and at the end. Reliability wording roughly doubles regex-hit verification events relative to speed wording, and three evaluations before repair make the first writable action more often an immediate modification than one evaluation does. On 128 historical trajectories, the treatment effect on correctness at the same repair point falls from 53.1 points to 6.25 points.
Critical strengths
Numerators and denominators are reported for every condition, and the correct counts at the first repair and at the end can be checked directly. But I don't think there are many strengths in this paper.
Critical weaknesses
The shrinkage of 46.9 points is the control group catching up to a treatment group already at the top. After the first repair LOW is 27/64 and HIGH is 61/64. At the end LOW is 57/64 and HIGH remains 61/64. The treatment group does not change, and the gap narrows because LOW uses up the remaining repair steps.
The remaining main results are the experimental procedure itself. The movement of the first action corresponds to one execution and three executions before repair. The verification doubling is 0.235 versus 0.472 regex hits per trajectory. Early verification events and terminal VERIFY are two different variables.
Questions for authors
- Besides HIGH stopping at 61/64, LOW filling from 27/64 to 57/64, and one execution versus three executions before repair, which result is still independent of this procedure?
- Are the correct counts for HIGH 61/64 at both checkpoints? If so, how many of these 46.9 points come from HIGH not declining?