Peer Review of Applied Machine Learning Papers Must Include Code Execution

This position paper argues that peer review of applied machine learning papers must include the execution and verification of submitted code. Reproducibility failures in ML are widespread, with documented cases spanning at least 17 scientific fields and hundreds of published studies. The current review system relies on a combination of voluntary code submission and self-reported checklists, leading to necessary post-publication correction and many wasted research hours. I support this argument with two case studies drawn from the EEG-based brain-computer interface literature, one in which the published claim is physically implausible and one in which available code runs cleanly but does not reproduce the reported numbers. I propose a structural change: a dedicated reproducibility reviewer with access to a containerized submission pipeline, applied uniformly to papers making empirical claims on real data.