Progress Rates Can't Tell Shortcuts from Progress: Landmark Properties for Scoring Agents

Declaring a step done is not doing it, yet agent benchmarks often score the two alike. Benchmarks give failing agents partial credit for checkpoints reached, but no rule in use asks how each checkpoint was reached. To our knowledge, no prior work states what such a score must satisfy. We supply that standard. From automated planning we take the order a task imposes on its milestones and state four properties a progress score needs. No score that reads only a run and its checkpoints can reliably detect skipped steps. The limit lies in the checkpoint list itself, not in the rule, and adding the milestone order removes it. We audit nine scoring rules, score every legal run in six task families, and test seven language models. The rules in use either ignore skipped steps or reward them. AgentBoard's published progress rate can rank a run that skips steps above a flawless one. Reanalyzing TheAgentCompany's released runs, we find that its final-checkpoint bonus alone produces 55 of 525 perfect scores. When an unlabeled shortcut appears among shuffled options, all seven models take it, and the published rate largely ranks them by how often they do. Our metric, weighted support, reverses that ranking. Given the milestone order and what each action required, it satisfies all four properties in one pass. Under weighted support, work that skips a prerequisite earns no credit, and neither does any work built on it.