SEA-Boundary: A Diagnostic Benchmark for State Preparation, Information Access, and Evidence Use in Long-Context LLM Systems

Limited context capacity motivates studying how LLM systems save information for later use and return to original documents when more detail is needed. We study how these systems prepare reusable memory (**State**), use it to guide requests for information from the documents (**Access**), and answer using State together with the retrieved passages (**Evidence**). When such a system answers incorrectly, the answer alone does not reveal whether the problem arose in State preparation, information access, or evidence use. We introduce SEA-Boundary, a controlled diagnostic benchmark for how knowledge prepared before questions guides the selection and interpretation of additional document facts. We specify what knowledge a model should retain and generate 96 synthetic document collections across four task families, deriving their questions and updates from the same facts and rules. Automated checks verify which combinations of retained knowledge and source records determine each answer, allowing new examples to be generated and checked. We compare independent function tests with complete workflows and inspect the memory and passages actually supplied in selected cases. Experiments show that correctly answering questions about State does not guarantee retrieving the facts needed to use that knowledge. Content checks identify cases where the relevant knowledge is correct in State but the model requests incomplete or irrelevant information. Inspecting actual answering inputs further distinguishes missing or incorrect information from mistakes made despite sufficient information. These tests evaluate memory through what it enables the model to read and answer, and help locate whether preparation, information access, or evidence use needs improvement.