: A Bounded Evaluation Protocol and Testbed for Representational Systematicity
Abstract
Systematicity is the capacity to generalize compositionally. Existing benchmarks primarily test behavioral systematicity, while representation-level analyses often leave open whether recovered signals are merely decodable or causally used during generation. We therefore introduce , an evaluation artifact and protocol for adjudicating these possibilities under bounded hypotheses: unsupported, present-only, causal-use, or underdetermined. The artifact is a controlled semantic-parsing setting with matched recombination tests, exact intervention pairs, and evidence routines for recoverability, interchange, necessity, and repair during generation. We apply to the two prominent representation-level explanations of systematic generalization, binding and geometry. Within the family instantiations evaluated here, and under the same behavioral anchor and state semantics, the binding family satisfies the criteria for , whereas the geometry family is adjudicated as under its frozen closure basis. Detector-truthfulness tests do not validate a geometry detector, so the geometry conclusion remains a claim about the tested family rather than about all possible geometric mechanisms. thus provides a reusable way to make disciplined representational claims from behavioral systematicity tests.
Related papers
See more related submissions when the market opens.