Externalising Knowledge for Decoupled Diagnosis: Mechanism-Level Attribution of Reasoning Failures

Benchmarks for large language models in chemistry and materials are built mostly from exam questions and terminology quizzes, scoring whether a model can state a known conclusion. Design work asks different queries – whether two mechanism statements denote the same concept, whether an effect holds under given conditions – which have no memorisable an- swer and can only be performed over a given knowledge structure. More fundamentally, failure cannot be attributed: when a model misses a mech- anism question, lacking the knowledge and failing to reason with it are indistinguishable, and that distinction decides whether the fix belongs to the knowledge source or to the model. We propose a design paradigm for diagnostic benchmarks: take the domain’s knowledge out of the model and write it as an executable structure – relations among concepts (composition, blocking, competition, and so on) as operations, axioms as inference rules, each item as an operator tree over that algebra. Two properties follow: answers are computed by executing the tree and hence recomputable; and the knowledge an item needs can be supplied tier by tier (withheld, com- plete, names only, or in a wrong order) with everything else fixed, making knowledge visibility a variable that can be manipulated on its own. The two tiers of one item separate “not knowing” from “not deriving”, and aggregat- ing such pairs over the items a knowledge unit gets wrong gives the unit’s compensability (the share of its failures external knowledge can rescue) – high-compensability units point at the knowledge source, low ones at the model. We propose a benchmark that can be diagnosed by attribu- tion, together with a paradigm for upgrading such ontology-based benchmarks: repair what the attribution diagnoses, and the di- agnosed unit’s performance improves. All readings are relative, and no absolute scale is claimed