acceptodds
Under review as a conference paper at ICLR 2027

SEA-Boundary: A Diagnostic Benchmark for State Preparation, Information Access, and Evidence Use in Long-Context LLM Systems

Abstract

Limited context capacity motivates studying how LLM systems save information for later use and return to original documents when more detail is needed. We study how these systems prepare reusable memory (**State**), use it to guide requests for information from the documents (**Access**), and answer using State together with the retrieved passages (**Evidence**). When such a system answers incorrectly, the answer alone does not reveal whether the problem arose in State preparation, information access, or evidence use. We introduce SEA-Boundary, a controlled diagnostic benchmark for how knowledge prepared before questions guides the selection and interpretation of additional document facts. We specify what knowledge a model should retain and generate 96 synthetic document collections across four task families, deriving their questions and updates from the same facts and rules. Automated checks verify which combinations of retained knowledge and source records determine each answer, allowing new examples to be generated and checked. We compare independent function tests with complete workflows and inspect the memory and passages actually supplied in selected cases. Experiments show that correctly answering questions about State does not guarantee retrieving the facts needed to use that knowledge. Content checks identify cases where the relevant knowledge is correct in State but the model requests incomplete or irrelevant information. Inspecting actual answering inputs further distinguishes missing or incorrect information from mistakes made despite sufficient information. These tests evaluate memory through what it enables the model to read and answer, and help locate whether preparation, information access, or evidence use needs improvement.

Primary area:
datasets and benchmarks
Keywords:
long-context language models, diagnostic benchmarks, knowledge retention, information access, evidence use

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NAVvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

test built to find exactly which step broke (memory, access, evidence) over synthetic fake world 2 types of qs, cheetsheet only, vs lookup systematic breakdown to analyze breakage of failure

Critical strengths
  • clear useful idea: helps inspect failure point and analyse
  • same wrong answer comes from different causes, needing different fixes
  • over HotpotQA
  • eval is factual and verified,
Critical weaknesses
  • doc with 22K tokens isnt long context anymore with current model capabilities (variation in length while testingshould be there)
  • benchmark is too easy (K3 gets 576/576 with correct inputs and 539/576 on its own)
  • real test pilot of 23 samples is thin
Questions for authors
3 comments

Thank you for the review, and for finding the idea clear and useful, the evaluation factual and verified, and the observation valuable that the same wrong answer can come from different causes needing different fixes. We respond to each weakness below; revisions are marked in blue in the PDF.

W2: "benchmark is too easy"

We agree that K3 is near ceiling overall: 539/576 planned tasks match completely (539/541 among tasks with an Answer output), and 576/576 with correct State and Evidence (Section 5.2; Table 2); the latter also confirms that every planned task is answerable as specified. Other configurations leave more headroom, for example 404/576 complete matches for DeepSeek max and 28/576 for Qwen3-30B-A3B (Table 12). We add three points:

  • Headroom and scaling. K3 is not at ceiling in every family: it matches 113/144 Purchasing tasks, whose requests follow retained mappings to item lists, bindings, and numerical records, and even with reference State its requests retrieve sufficient Evidence for 133/144 (Table 8). It fully solves the longest family (about 22K tokens), so difficulty here varies with dependency structure rather than length. The 96 worlds are the evaluated core, not a generator ceiling: new seeds add checked instances, and new families with their own dependency structures can be added under the same protocol and checks (Section 4.2).

  • Smaller models. The benchmark separates distinct limitations below the frontier (Appendix B.4; Table 10). With correct State supplied, Falcon-H1-7B answers 559/1,182 State-only questions but selects sufficient Evidence for only 66/576 tasks, and Qwen2.5-1.5B answers 422/1,182 but selects sufficient Evidence for none. Llama-3.1-8B instead fails mainly at the output interface (612 invalid-syntax and 379 invalid-shape recall outputs of 1,182). These call for different repairs.

  • Strong models. Near ceiling, the benchmark locates the remaining failures. K3's 37 are 35 tasks without an original Answer call, all answered correctly once correct inputs are supplied, and two answered nonmatches. For example, an inspected K3 State binds a task to the wrong production line; Access fetches that line's genuine prices and Answer sums them correctly, so the repair target is the binding. In contrast, a Qwen3.8-27B answer uses another service's lead time although State and Evidence contain the right offer (Section 5.4). Replacing only defective Evidence at Answer changes four selected DeepSeek tasks from 0/4 to 4/4 matches under a matched-format, thinking-disabled baseline, with Access unchanged (Table 14).

W1: "doc with 22K tokens" and "variation in length"

We agree that 22K tokens fits within current context windows. The benchmark targets a different question: whether memory prepared before the questions are known later guides source selection and evidence use. Stage-level failures, including omissions with the required knowledge in State, already appear with sources of 5,797 to 21,949 tokens across four families (Table 4; Section 5.3). In this range, K3 fully solves the longest family while missing tasks in shorter ones (Table 8).

To measure length effects directly, we plan a controlled comparison that adds source records while holding task dependencies fixed, so each stage can be tracked as length grows. The current lengths vary with task family, and this design separates the two.

W3: "real test pilot of 23 samples is thin"

The synthetic core carries the main evidence because it supplies what this diagnosis needs and natural corpora rarely provide: known retention requirements and verified support records for each answer (Section 4). We agree that the 23-question natural pilot is narrow; it tests whether one mechanism carries over. With reference State, State-guided BM25 yields 21/23 source-supported DeepSeek answers versus 4/23 for question-only retrieval with State supplied at Answer. It uses unchanged HotpotQA passages, one film-director template, six shared source groups, 557 to 909 tokens, and no Update; both methods retrieve two paragraphs, with different passages and token counts (Appendix D).

The new batch is designed for breadth across text types rather than count: the same protocol runs on Wikipedia multi-hop questions (HotpotQA, 2Wiki), scientific papers (QASPER, 5K to 8K tokens), and meeting transcripts (QMSum, 14K to 17K tokens), one preselected question per source, with papers and meetings read as whole units. With and without question-blind model summaries, all answers were correct with source support. Under the same catalog and reading budget, three summaries shortened lookup paths, one enabled a direct answer, and four pairs read equally. We will extend this batch to more sources per dataset, including section- and turn-level reading in papers and meetings, and report it separately with summary-construction costs.

Thank you again; we would be glad to answer any further questions.

AuthorsOpenReview ↗

Follow-up on W1, W2, and W3

Thank you again for your review. Building on our first response, we clarify the stage inputs for W1, add a Purchasing diagnostic for W2, and report the completed natural-text extension for W3.

W1: source length and variation in length

One clarification concerns what each stage receives. Build reads the full source before questions are revealed; readout and Access later receive State and the question. Answer receives State, the question, and retrieved Evidence. Any required support not retained in State must therefore be recovered through Access (Section 3). This tests whether knowledge prepared before the questions guides later source selection, even when the source would fit within a context window. Purchasing's initial sources range from 9K to 19K tokens (Table 4).

W2: "benchmark is too easy"

In the original evaluation, Kimi K3 answers all 300 Purchasing State-only questions correctly from its own State (Section 5.2; Table 8). Our first response reported its 113/144 complete workflow matches and, in a separate test with reference State supplied to Access, sufficient Evidence for 133/144 tasks. Together these results distinguish retained knowledge, source access, and complete workflow performance: correct supplied knowledge does not guarantee successful source access. Measuring request execution and Evidence sufficiency alongside readout reveals a repair target that final-answer accuracy alone does not separate.

W3: completed natural-text extension

As planned in our first response, we extended reading to paper sections and meeting turns. The study covers 16 source/question pairs: three multi-hop Wikipedia contexts, seven scientific papers, and six meeting transcripts of 5.3K to 26.7K tokens. Both conditions used gpt-6-astra/xhigh with the same available source text and reading allowance.

  • Answer content and format: independent agent review found all 32 final texts complete and supported by the inputs each condition saw. Five JSON-format failures remain failures.
  • Reading and construction costs: the summary + semantic catalog condition read 44,254 source tokens versus 127,012 for opaque chunk IDs only (65.2% less). Including recorded construction once, total API use was 277,371 versus 179,681 tokens.

These results concern the combined summary/catalog at unequal retained capacity, with one trajectory per condition. Our follow-up general response and Appendix G.2 of the revised PDF (Table 22) give the protocol, all costs, and the construction costs of the earlier eight-question batch.

Since the discussion period closes soon, we would welcome your feedback on whether these results and clarifications address your concerns, and on any points that still need attention.

AuthorsOpenReview ↗

Follow-up on W1 and W2: source-length and task-difficulty studies

Thank you again for your review. In our first response we committed to a controlled comparison that adds source records while holding task dependencies fixed. It is now complete (W1), together with a second study that makes tasks harder by making each answer depend on more source records (W2). Both are reported in Appendices G.3-G.4 of the revised PDF, and the supplementary material now includes their execution evidence: every request, output, retrieved Evidence, and score.

W1: the original tasks extend to 100K-token sources. We extended two original Assembly worlds from about 10K to 32K and 100K tokens by adding stock records for further production runs, generated under the original business rules and queryable through Access. Original documents, questions, and updates are unchanged, and the original validation checks are retained. The State limit stays at its original absolute size (about 1,520 tokens), about 1.5% of the 100K source, and Build reads the entire source before any question is shown. All 72 Access requests execute under exact name matching, and all 24 States stay within the limit. DeepSeek V4.1 Flash-1017 answers 10/12, 12/12, and 12/12 correctly at the three lengths; Kimi K3.1 Pro answers 12/12 at each. Flash-1017's two failures at the shortest length trace to one State error of the kind in the K3 example of Section 5: the State assigns a coordinator to the wrong production line, so Access retrieves genuine prices from that line; two State-only questions also expose the error. The original tasks thus run on a tenfold longer source with their validation checks retained, and where a failure occurs, the stage measurements place it in State rather than in Access or Answer.

W2: harder tasks expose failures that the diagnostics locate. In a separate study on two regenerated Assembly worlds, each answer depends on the records of 3, 6, or 9 components (the parts combined into a kit), while the source stays at about 21.4K tokens and the State and Access limits are fixed. Flash-1017 answers 12/12 at every setting; K3.1 Pro answers 12/12, 12/12, and 6/12. All six errors come from one State that assigns three coordinators to the wrong production lines, an error kept through Update; Access executes, and the answers follow correctly from the wrong lines' records. Two of these answers still match the reference final values while their per-component numbers differ, so a score that ignores per-component numbers counts 8/12, while complete-answer scoring counts 6/12. We do not claim that the number of components caused this error. The point is that the stage measurements trace a strong model's failures to the wrong line assignments in one State, which a task-level score partly hides.

Both studies use Flash-1017 and K3.1 Pro on two Assembly worlds and build and update one State per world, model, and setting, so we compare results only within each study; the original 96-world scores are unchanged. Together with our earlier replies on Kimi K3's Purchasing results (W2) and the 16-pair natural-text study (W3, Appendix G.2), we hope this addresses your concerns, and we would be glad to clarify anything further.

Reviewer_FF6FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SEA-Boundary, a diagnostic benchmark that separates preparing reusable State, selecting information through Access, and answering with State plus retrieved Evidence. Its motivation is that final-answer accuracy cannot identify whether failure originates in retained knowledge, information selection, or evidence use. The benchmark contains 96 synthetic document worlds across four task families and two rounds. Reference-input controls and selected audits of actual inputs support more precise attribution. A useful observation is that correctly answering a recall question does not guarantee successful evidence selection.

Critical strengths

Pros

  1. The diagnostic distinctions are useful for system development. Separating executable requests, sufficient retrieved evidence, sufficient actual joint inputs, and correct evidence use provides substantially more guidance than a single accuracy score.

  2. The generated tasks have inspectable information dependencies. Rules, source records, updates, and reference answers come from a shared construction process, supported by independent replay, deletion checks, and equivalent-support checks.

  3. The paper is careful about what constitutes a semantic failure. It distinguishes truncation, invalid formats, missing calls, ambiguous intermediate fields, and genuine answering errors under sufficient inputs.

Critical weaknesses

Cons

  1. External validity remains limited. The main tasks are synthetic, explicitly specify what knowledge should be retained, and permit only one Access request before evidence is returned.

  2. Interface and scoring effects complicate interpretation of the benchmark scores. Length limits, operation constraints, and formatting failures substantially affect some configurations.

Questions for authors

Questions

  1. Is the recall–selection gap primarily about missing content or the representation of correct content?

  2. How stable are the conclusions when Access becomes iterative?

3 comments

Thank you for the thoughtful review, and for finding the diagnostic distinctions useful for system development, the information dependencies inspectable, and the treatment of semantic failures careful. We answer each point below; revisions are marked in blue in the PDF.

Q1: "Is the recall–selection gap primarily about missing content or the representation of correct content?"

Both occur, and the gap also appears when the required knowledge is present. In the supplier example, State retains the buyer zone, review panel, threshold, waiver, and scope, yet Access retrieves nine of the twelve required scores, although three allowed operations can retrieve all four suppliers (Section 5.3). The omission arises in constructing the request.

Across the cohort, 56 of 192 update-dependent Qwen3-30B-A3B tasks have a correct State-only readout; 34 of their requests execute in replay and 11 return Evidence sufficient with reference State. Of the remaining 23, the original inspection found task-relevant State correct in four and defective in 19 (Appendix C.4).

Three of the four entered the blinded re-evaluation. All judges label every task-relevant State axis correct in two. The third shares State with the supplier example: judges confirm its routing and policy values but flag a summary that dates all records to September 5 despite a September 6 revision; this does not change the policy for the September 6 request (DeepSeek also labels it a content defect). The cohort therefore contains both kinds of failure: 19 omissions with State defects, and four with the required knowledge present.

Q2: "How stable are the conclusions when Access becomes iterative?"

We ran an iterative pilot on eight archived tasks fixed in advance (seven worlds): the four selection examples, two with recorded State defects, and two original successes. Using their saved States, a different model, DeepSeek V4.1 Flash-1017, answered 8/8 correctly in each of three conditions: one request under the original allowance; up to two requests chosen before seeing Evidence; and an optional second request after seeing the first Evidence, with the same maximum total allowance as the second condition. Every optional second request was empty, because the first requests already returned sufficient Evidence.

Adding iteration therefore left accuracy unchanged on these tasks. The next step is to start from first requests that omit records, as in the supplier example, and measure whether the model adds them after seeing Evidence.

W2: "Interface and scoring effects complicate interpretation of the benchmark scores."

We agree that these effects shape some configurations' scores. The reported results keep them, and the diagnostics attribute each to its source:

  • Length limits. All 22 calls in a selected Qwen3.8-27B Access set hit the length limit; with identical inputs and a fourfold allowance, all 22 new requests execute and return Evidence sufficient with reference State (Appendix E.1).

  • Operation constraints. All records required in the supplier example fit within three allowed operations, so that omission reflects request construction.

  • Format and interface. The historical implementation rejected 50 saved requests whose literal-array selections the public rules allow; they execute in replay and are reported as interface noncompletions, with no Answer call (Appendix B.1).

  • Scoring. The original audit classed 19 of the 39 completed, format-valid mismatches as ambiguous intermediate-field cases with correct task results, because capacity units or margin formulas were not separately defined (Section 5.4); Table 2 therefore reports both complete-answer and task-result scores. For the 16 re-evaluated cases, accepted reviews (Kimi 16, Grok 14, DeepSeek 1) mark final results correct and capacity entries incorrect under the kit-capacity reading, which source checks support, while the output-field unit remains implicit. The revision reports this refinement; the scores in Table 2 are unchanged.

W1: "External validity remains limited."

The synthetic tasks specify the knowledge to retain and expose source dependencies, which is what allows preparation and selection to be checked separately; our main claims are stated for this controlled workflow (Section 6). The one-request setting is examined by the iterative pilot above.

Beyond the original 23-question pilot (Appendix D), a new batch of eight questions from four datasets compared access with and without question-blind model summaries. All answers in both conditions were correct and source-supported; summaries shortened three lookup paths and enabled one direct answer, and four pairs read equally. We will report it separately, with summary-construction costs.

Thank you again; we would be glad to discuss any further questions.

Thanks for your response, which has addressed my concern. I prefer to keep my score with the weak acceptance.

Thank you for the follow-up and for confirming that our response addressed your concern. We appreciate your careful assessment.

Reviewer_qnK3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SEA-Boundary, a benchmark for diagnosing three stages in long-context LLM workflows: preparing a compact memory (State), using it to request source records (Access), and answering from the memory and retrieved evidence. It includes 96 synthetic document collections across four task families, with controlled updates and checks of which records are needed for each answer. Experiments show that a model may recall the right information but still fail to retrieve all the records needed for a later task.

Critical strengths
  1. The benchmark defines the information needed for each answer and checks it automatically. This makes the retrieval results more interpretable than final-answer accuracy alone.
  2. The paper shows that similar wrong answers can have different causes, such as incorrect memory, missing retrieved records, or errors despite sufficient inputs.
Critical weaknesses
  1. Most experiments use synthetic tasks with structured records and a fixed retrieval interface. The natural-text test has only 23 questions from one template, so it provides limited evidence that the findings generalize to realistic long-context tasks.
  2. The input audit covers only 113 of 2,358 nonempty answers, using one LLM auditor per case without agreement measurements. This makes the error categories less reliable than the paper’s broader discussion may suggest.
Questions for authors
  1. Can the authors validate the input-sufficiency judgments with human annotations or multiple independent reviewers and report agreement?
  2. Can the authors add a broader natural-text evaluation, or narrow the claims to the controlled synthetic setting?
  3. How would the results change under a matched State/token budget? The current configurations use different generation limits and do not penalize extra retained facts.
5 comments

Thank you for the careful review, and for noting that the benchmark defines and automatically checks the information needed for each answer, and that similar wrong answers can have different causes. We respond to each point below; revisions are marked in blue in the PDF.

W2 and Q1: "one LLM auditor per case without agreement measurements"

We added a blinded multi-judge re-evaluation, now in Appendix C.1. Three model judges (DeepSeek, Kimi, and Grok) re-evaluated 101 of the 113 originally inspected inputs, from 40 worlds; the other 12 were used to develop the protocol. Each judge assessed State and Evidence against source references without the original label, and locked this input judgment before seeing the Answer. Judges skipped answers from their own model family.

Every valid input judgment reproduces the original label: DeepSeek 59/59, Kimi 73/73, and Grok 94/94, including 28, 30, and 28 insufficient-input cases. Valid input reviews cover 59/61, 73/73, and 94/101 eligible cases, and complete two-stage reviews cover 58/61, 72/73, and 93/101; failed calls stay in these denominators (one Kimi answer-stage call hit a provider usage limit and was not retried).

Main answer attribution agrees in 28/29 DeepSeek-Kimi, 65/66 Kimi-Grok, and 50/54 DeepSeek-Grok matched complete reviews, and in 26/27 across all three. Four DeepSeek-Grok differences concern intermediate-capacity readings: both judges find the final results correct, and source checks support the answers' kit-capacity reading (Kimi skips these by model family). In the fifth, all three find insufficient inputs and an incorrect result and differ only on whether a format failure takes priority.

The original audit inspected 113 of 2,358 nonempty Answers, including all 39 completed, format-valid mismatches and all 23 readout-selection cases (Appendix C.2); in these two cohorts, one auditor per case saw the outcomes. The new agreement supports the labels in this audited cohort, which the paper uses for case-level attribution rather than benchmark-wide rates. The judges also refine two finer labels, summarized in our general response.

The benchmark scores need no semantic judgment: answer scores, request executability, and Evidence sufficiency with reference State come from deterministic scoring and replay. Semantic inspection is needed only for actual-input sufficiency, which assesses the model-written State plus Evidence, including extra facts and scope.

W1 and Q2: "add a broader natural-text evaluation, or narrow the claims to the controlled synthetic setting"

We adopt the scope you suggest: the main findings are stated for the controlled synthetic workflows (Abstract; Sections 1 and 6). The natural-text pilot tests one mechanism, whether retained relations improve passage selection. With reference State, State-guided BM25 yields 21/23 source-supported DeepSeek answers versus 4/23 for question-only retrieval with State supplied at Answer (Appendix D). It uses unchanged HotpotQA passages, one film-director template, six shared source groups, 557 to 909 source tokens, and no Update; both methods retrieve two paragraphs, with different passages and token counts.

To broaden the evidence, we ran eight questions from HotpotQA, 2Wiki, QASPER, and QMSum, selected before generating question-blind model summaries. With and without summaries, all eight answers were correct with source support. Under the same catalog and reading budget, summaries shortened three lookup paths and enabled one direct answer; four pairs read the same amount. We will report this batch, including summary-construction costs, separately from the original pilot.

Q3: "How would the results change under a matched State/token budget?"

The readout-selection gap is measured within each State, so it does not depend on budget differences between configurations: readout and Access receive the identical model-written State. Of 192 update-dependent Qwen3-30B-A3B tasks, 56 have a correct readout; 34 of their requests execute in replay, and 11 return Evidence sufficient with reference State (Section 5.3). Reference-input controls likewise hold reference State fixed across configurations.

A matched budget would test how often such gaps occur when State must be compressed to a common size. This comparison should fix final retained-State tokens separately from generation allowances, which include reasoning where enabled (Section 3.1; Appendix B.1).

Extra correct facts count toward actual-input sufficiency because Answer can use them; ignoring them could misclassify an answering error as missing information. State length is reported separately (Tables 1 and 7), so retention efficiency can be compared directly.

Thank you again for the concrete suggestions; we would be glad to clarify any remaining point.

Thanks to the authors for the response. Most of my concerns have been addressed, and I decide to raise my score to 3.

Thank you for the follow-up and for confirming that our response addressed most of your concerns.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reviews. They found the core idea clear and useful (NAVv), the information required for each answer explicit and automatically checked (qnK3, FF6F), and the separation of different causes behind the same wrong answer valuable for fixing systems (qnK3, FF6F, NAVv). The main concerns are the reliability of the input audit, iterative Access, natural-text breadth, benchmark difficulty, and source length. Below we summarize the new evidence; each response gives the details.

Blinded re-evaluation of the input audit

Three model judges (DeepSeek, Kimi, and Grok) re-evaluated 101 of the 113 originally inspected inputs, from 40 worlds; 12 further cases were used to develop the protocol. Each judge assessed State and Evidence against source references before seeing the Answer, without the original label, and skipped answers from its own model family.

Every valid input judgment matches the original audit: 59/59, 73/73, and 94/94, respectively. Main answer attribution agrees in 28/29 DeepSeek-Kimi, 65/66 Kimi-Grok, and 50/54 DeepSeek-Grok matched pairs. Coverage, the five attribution differences, and one missing assessment are detailed in our response to qnK3.

Iterative Access

On eight archived tasks fixed in advance, including the four selection examples, a different model (DeepSeek V4.1 Flash-1017) used the saved States in three conditions: one request, up to two requests chosen before seeing Evidence, and an optional second request after seeing the first Evidence. All three conditions answered 8/8 correctly, and every optional second request was empty because the first requests already returned sufficient Evidence (response to FF6F).

Natural-text breadth and scope

The main claims are stated for the 96 checked synthetic worlds in four families, and the generator produces further checked instances in each family (Section 4.2). The original natural-text pilot tests State-guided retrieval on 23 HotpotQA questions from one template (Appendix D).

We add eight preselected questions from HotpotQA, 2Wiki, QASPER, and QMSum, answered with and without model-written summaries prepared before the questions were seen. Both conditions answered all eight correctly with source support. Under the same catalog and reading budget, summaries shortened three lookup paths and enabled one direct answer; four pairs read the same amount. Summary construction is an additional cost, which we will report with this batch.

What the diagnostics add

Readout (answering from State alone) and Access receive the identical model-written State, so successful recall and incomplete retrieval can be told apart. In a supplier task, State contains the required routing and policy knowledge, yet Access fetches nine scores for three suppliers when twelve scores for four are needed, although all twelve fit within the allowed operations (Section 5.3; Appendix C.4). The repair target is request construction.

Completed controls show that different failures call for different repairs. On four selected DeepSeek tasks with correct State but defective Evidence, replacing only Evidence at Answer changes a matched-format, thinking-disabled baseline from 0/4 to 4/4 matches (Table 14). Given correct inputs, DeepSeek max correctly answers 151 of 159 tasks whose original workflows made no Answer call (Section 5.2). The same checks apply to smaller models and the other tested architectures.

Revised PDF

The revision reports the blinded re-evaluation (Section 3.3; Appendix C.1) and two refinements it brings: 16 intermediate-field cases that the original audit left ambiguous are labeled use errors under the kit-capacity reading, and one State shared by two selection examples has a misdated summary (Sections 5.3 and 5.4; Appendices C.2 and C.4). Changes are marked in blue; reported scores are unchanged.

AuthorsOpenReview ↗

Follow-up general response: new results and revised PDF

Follow-up general response: new results and revised PDF

We report completed results for the concerns below and summarize the revised PDF.

qnK3 Q3: matched State/token budgets and extra retained facts

We ran a supplementary shared-budget study with DeepSeek V4.1 Flash-1017 and Kimi K3.1 Pro, different model variants from the original DeepSeek V4 Pro and Kimi K3. Under the same generation and State-size limits within this study, they succeeded on 36/40 and 39/40 task chains, respectively (Appendix G.1, Table 21).

Both used the same 20 stress-case worlds, selected because States in the original API runs exceeded 30% of source (19 Service, 1 Purchasing). Each world contributes a Build-round and an Update-round task, giving 40 task chains per model. Both received a 16,384-token generation limit, a State target of 10% of source, and a 15% hard cap. The entire State was counted, including extra facts, formatting, and provenance. Extra correct facts are allowed, count toward the cap, and remain available to Answer.

All 79 normal nonempty States met the cap; median State/source ratios were 9.660% for DeepSeek and 9.669% for Kimi. Four tasks failed because outputs omitted the "service" suffix of names although the States retained the policy orders and thresholds; no Answer was called. One DeepSeek Update used its 16,384-token allowance for reasoning without producing a State. All five failures remain in the denominators.

These results show that the two supplementary models succeeded on most task chains with States near 10% of source on this selected panel. The original 96-world scores retain their original model variants and budget settings.

qnK3 W1/Q2, NAVv W3, and FF6F W1: broader natural-text evaluation

We report construction costs for our earlier eight-question batch and a broader study with section- and turn-level reading. Because their panels and contrasts differ, we report them separately.

For the earlier batch, which varied summaries under a fixed catalog, construction used 62,701 API tokens; reading used 55,351 with summaries versus 61,054 without. Charging construction once gives 118,052 versus 61,054.

The new study asks whether a question-blind summary and semantic catalog help a reader choose where to read. It uses 16 existing source/question pairs: two HotpotQA contexts, one 2WikiMultihopQA context, seven QASPER papers, and six QMSum meetings. Its two initial conditions are:

  • A: summary + semantic catalog, with the summary prepared without seeing the question.
  • B: chunk IDs only, the same opaque chunk IDs without the summary or semantic descriptions.

Both conditions used gpt-6-astra/xhigh, the original questions and the same available source text. Each could read any chunks or the full source, with two Reads and at most three decisions; neither had keyword or semantic search. Summaries were reused unchanged. Protocol and aggregate results appear in Appendix G.2, Table 22.

  • Answer content: independent agent review found all 32 final texts complete and supported by the inputs each condition actually saw.
  • Interface: normal JSON answers were 14/16 for A and 13/16 for B; all five format failures remain failures despite complete natural-language content.
  • Reading costs: A returned 44,254 source tokens versus B's 127,012 (65.2% less), and used 127,604 versus 179,681 reader API tokens (29.0% less).

Three examples illustrate these differences. For a paper's experiment-list question, A read four relevant blocks (949 tokens) versus B's full available paper text (1,832); both covered all requested experiment types, although A's reader API use was slightly higher. For a teacher-training question, A read five meeting blocks (2,757 tokens) versus B's 11,682, and both preserved the witnesses' different positions. For meeting-recording conventions, both read all 18,027 tokens. Full reading was a model choice, not imposed on B.

Reading costs and construction costs are reported separately. Charging the recorded 149,767 construction tokens once gives 277,371 total API tokens for A, above B's 179,681. The full retained summary plus catalog occupied 12.3% to 132.3% of source. These results concern the combined summary/catalog at unequal capacity on the specified existing pairs, with one trajectory per condition.

Revised PDF

Appendix G reports shared budgets in G.1 (Table 21) and natural-source reading with both cost comparisons in G.2 (Table 22). Sections 3 and 6 limit the absence of a shared State cap to the original runs; Section 6 separates the original natural-text pilot from G.2; Section 5.2 adds K3's Purchasing readout, workflow and reference-State Access counts. Changes are marked in blue; original reported scores are unchanged.

AuthorsOpenReview ↗

Summary for the AC: paper, reviewer assessments, discussion, and revisions

The paper. SEA-Boundary is a controlled diagnostic benchmark, evaluated on 96 checked synthetic document collections in four families, for workflows that prepare memory (State) before questions are known, use it to request source records (Access), and answer from State plus the returned Evidence. Construction checks fix which knowledge and records determine each answer, so a failure can be traced to the stage where it arises. The central finding is that remembering is not the same as knowing what to look up: a model can answer correctly from its State yet fail to request the records a later task needs.

Strengths noted by reviewers. All three reviewers value the stage-level diagnosis: the same wrong answer can come from different causes that need different fixes (qnK3, FF6F, NAVv). They also note automatically checked information needs (qnK3, FF6F), a factual, verified evaluation (NAVv), and more guidance than a single accuracy score (FF6F).

Discussion outcome.

  • qnK3 asked about audit agreement, natural-text breadth, and matched State budgets. qnK3 replied on 17 November that most concerns were addressed and raised the score to 3; the shared-cap and natural-text studies were posted the next day. Our final follow-up adds the original models to the shared-cap study and corrects one supplementary score from 36/40 to 37/40 after rerunning a truncated output.
  • FF6F asked whether the recall-selection gap reflects missing content or how correct content is represented, and how iterative Access affects the conclusions. We showed request omissions both with State defects and with the required knowledge present (Appendix C.4), and reported an iterative-Access pilot: 8/8 with or without a second request, as first requests already sufficed. FF6F replied that its concern was addressed and kept weak acceptance.
  • NAVv raised source length (W1), difficulty (W2), and the 23-question natural pilot (W3). We responded on 15 and 18 November and posted a final follow-up today; NAVv has not yet replied. On W1, once State is built, later stages see only State, the question, and (for Answer) retrieved Evidence, so the benchmark tests whether knowledge prepared from the full source guides later access. On W2, K3 is near ceiling overall but not in every family: it answers all 300 Purchasing questions from State alone yet completes 113/144 Purchasing workflows. The studies below address all three points.

Revisions and what they address (in blue; original 96-world scores unchanged).

  • Label reliability (qnK3; Section 3.3, Appendix C.1). Three blinded model judges re-evaluated 101 of the 113 audited cases and reproduced every valid input-sufficiency label, so the labels do not hinge on one judge.
  • Matched State budget (qnK3; G.1). On 20 worlds whose original States exceeded 30% of source, with a 15% cap on the complete State, Kimi K3, DeepSeek V4 Pro, DeepSeek V4.1 Flash-1017, and Kimi K3.1 Pro answer 35, 35, 37, and 39 of 40 tasks (V4 Pro 34/40 with all States in cap). V4 Pro: 36/40 at a median State of 31.5% of source in the paper, 35/40 at 11.1% here. The failures are names missing a required suffix or answers calling sufficient inputs insufficient, which the stage measurements separate.
  • Natural text (qnK3, FF6F, NAVv W3; G.2). 16 question-source pairs from Wikipedia multi-hop contexts, scientific papers, and meetings, each run in two setups. All 32 answers have complete, source-supported content (five JSON-format errors count as failures). A question-blind summary with chunk descriptions cut source tokens read by 65.2% versus bare chunk IDs but used more API tokens once summary preparation is counted (unequal retained capacity; one run each). Main claims stay scoped to the synthetic setting (Abstract; Sections 1, 6).
  • Source length (NAVv W1; G.3). Two original Assembly worlds extended from about 10K to 32K and 100K tokens, keeping the validation checks and the absolute State limit (about 1.5% of the 100K source). All 72 Access requests execute; Flash-1017 answers 10/12, 12/12, and 12/12, and K3.1 Pro 12/12 at each length; the two failures, at the shortest length, trace to one wrong line assignment in a State. The original tasks thus remain executable on a tenfold longer source.
  • Task difficulty (NAVv W2; G.4). In two regenerated Assembly worlds of about 21.4K tokens, with 3, 6, or 9 required components, Flash-1017 answers 12/12 at every setting and K3.1 Pro 12/12, 12/12, and 6/12. All six errors trace to wrong line assignments in one State, and a score ignoring per-component arithmetic would count 8/12. The stage measurements thus locate failures that a task-level score partly hides; we do not claim the component count caused them. Both use one run per condition.
  • Supplementary material (NAVv W1-W2): the original files plus every G.3-G.4 request, output, Evidence, and score.

We thank the AC and reviewers for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.