A Verifiable Environment for Agents That Design Neural Architectures

Reinforcement learning and agent evaluation both need a verifier: a function that scores a candidate cheaply, with no human in the loop. While software development relies on compilers and mathematics on automated answer checkers, neural architecture design has lacked a deterministic, public verifier. We present an environment in which an agent designs a neural network by editing a typed graph, and a deterministic verifier grades the result in under a millisecond against structural, budget, and serving-cost constraints. No human and no LLM takes part in scoring. The environment includes a procedural task generator (ten families, with satisfiability and non-vacuity checks and red-teamed anti-gaming rules), a difficulty-calibration harness, and a grounding study. In that study, across $264$ architectures, the verifier blocks every corrupted graph ($184/184$), no blocked graph survives a real PyTorch forward pass ($0/184$), and every clean graph runs ($80/80$). The $0$ to $100$ health score is a validity margin, partly a size proxy, and not a quality ranking. We report four findings. As a repair signal, verifier feedback lifts claude-sonnet-4-6 from $67\%$ to $93\%$, grok-4 from $78\%$ to $96\%$, and deepseek-chat from $48\%$ to $93\%$; the lift concentrates on arithmetic defects, while cross-branch topology is largely unfixed. As a correction to how design agents are scored, a rubric that does not propagate shapes rates the same models $13$ and $14$ points higher than they are. As a training-data generator, fine-tuning a 1.5B model on $3,000$ verified pairs lifts pass@1 from $17\%$ to $76\%$ on held-out tasks from the same ten families, and $100$ GRPO steps on top reach $80\%$ ($n=192$). Under a stricter rubric the same recipe runs $15\%\to 70\%\to 80\%$, and a 7B policy starts lower than the 1.5B ($5\%$, because it is more verbose) and ends at $76\%$. Transfer is not general: four families held out of training collapse to $0\%$. A contamination audit we ran on ourselves shows the naive protocol would have reported $86\%$. As ground truth, eleven LLM reward models have a $0\%$ false-positive rate on blatantly broken designs (ten of eleven when re-measured), yet approve $24$ to $48\%$ of near-miss designs that are one edit from correct. The verifier is $0\%$ on both tiers. The same function serves as a leaderboard, an RL gym, and a verified-data generator. We release all of it.