From Performance to Capability: A Three-Level Framework for Behavioral LLM Capability Science with Large-Scale Empirical Study

Behavioral LLM capability science asks how evidence from observable behavior can justify increasingly strong claims about what an LLM system can do and how its behavior is organized. We propose a three-level framework that distinguishes (1) Observable Performance, (2) Conditional Response and Capability Validity, and (3) Fundamental Organization, and makes the evidential transitions between them explicit and testable. We evaluate this framework across 34 heterogeneous released source collections spanning reasoning, feedback, memory, prompting, planning, coding, retrieval, and related settings, comprising over one million task-level responses. Conditional Response emerges as a substantive empirical object: response profiles reproduce across compatible roots (rank correlation 0.586), 73/311 eligible groups exhibit held-root nonconstant regularities, and multiple designs show nonadditive effects. Stronger capability claims require connected evidence involving reliability, transfer, convergence across operationalizations and sources, specificity, discrimination, measurement quality, controls, external prediction, and compatible common support. Four of nine candidate constructs obtain local reliability and task-transfer support, and one obtains cross-operationalization support, but none completes connected validity because key links in the released evidence topology remain missing or conflicting. Human-readable semantic and conditional structure is predictively useful on held-out data, yet predictive or low-dimensional structure alone does not establish Fundamental Organization. The resulting evidence frontier distinguishes observed performance, established response organization, supported capability evidence, and unresolved deeper organization, providing a concrete foundation for cumulative LLM capability science.