- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
test built to find exactly which step broke (memory, access, evidence) over synthetic fake world 2 types of qs, cheetsheet only, vs lookup systematic breakdown to analyze breakage of failure
Critical strengths
- clear useful idea: helps inspect failure point and analyse
- same wrong answer comes from different causes, needing different fixes
- over HotpotQA
- eval is factual and verified,
Critical weaknesses
- doc with 22K tokens isnt long context anymore with current model capabilities (variation in length while testingshould be there)
- benchmark is too easy (K3 gets 576/576 with correct inputs and 539/576 on its own)
- real test pilot of 23 samples is thin
Questions for authors
3 comments
Follow-up on W1, W2, and W3
Thank you again for your review. Building on our first response, we clarify the stage inputs for W1, add a Purchasing diagnostic for W2, and report the completed natural-text extension for W3.
W1: source length and variation in length
One clarification concerns what each stage receives. Build reads the full source before questions are revealed; readout and Access later receive State and the question. Answer receives State, the question, and retrieved Evidence. Any required support not retained in State must therefore be recovered through Access (Section 3). This tests whether knowledge prepared before the questions guides later source selection, even when the source would fit within a context window. Purchasing's initial sources range from 9K to 19K tokens (Table 4).
W2: "benchmark is too easy"
In the original evaluation, Kimi K3 answers all 300 Purchasing State-only questions correctly from its own State (Section 5.2; Table 8). Our first response reported its 113/144 complete workflow matches and, in a separate test with reference State supplied to Access, sufficient Evidence for 133/144 tasks. Together these results distinguish retained knowledge, source access, and complete workflow performance: correct supplied knowledge does not guarantee successful source access. Measuring request execution and Evidence sufficiency alongside readout reveals a repair target that final-answer accuracy alone does not separate.
W3: completed natural-text extension
As planned in our first response, we extended reading to paper sections and meeting turns. The study covers 16 source/question pairs: three multi-hop Wikipedia contexts, seven scientific papers, and six meeting transcripts of 5.3K to 26.7K tokens. Both conditions used gpt-6-astra/xhigh with the same available source text and reading allowance.
- Answer content and format: independent agent review found all 32 final texts complete and supported by the inputs each condition saw. Five JSON-format failures remain failures.
- Reading and construction costs: the summary + semantic catalog condition read 44,254 source tokens versus 127,012 for opaque chunk IDs only (65.2% less). Including recorded construction once, total API use was 277,371 versus 179,681 tokens.
These results concern the combined summary/catalog at unequal retained capacity, with one trajectory per condition. Our follow-up general response and Appendix G.2 of the revised PDF (Table 22) give the protocol, all costs, and the construction costs of the earlier eight-question batch.
Since the discussion period closes soon, we would welcome your feedback on whether these results and clarifications address your concerns, and on any points that still need attention.
Follow-up on W1 and W2: source-length and task-difficulty studies
Thank you again for your review. In our first response we committed to a controlled comparison that adds source records while holding task dependencies fixed. It is now complete (W1), together with a second study that makes tasks harder by making each answer depend on more source records (W2). Both are reported in Appendices G.3-G.4 of the revised PDF, and the supplementary material now includes their execution evidence: every request, output, retrieved Evidence, and score.
W1: the original tasks extend to 100K-token sources. We extended two original Assembly worlds from about 10K to 32K and 100K tokens by adding stock records for further production runs, generated under the original business rules and queryable through Access. Original documents, questions, and updates are unchanged, and the original validation checks are retained. The State limit stays at its original absolute size (about 1,520 tokens), about 1.5% of the 100K source, and Build reads the entire source before any question is shown. All 72 Access requests execute under exact name matching, and all 24 States stay within the limit. DeepSeek V4.1 Flash-1017 answers 10/12, 12/12, and 12/12 correctly at the three lengths; Kimi K3.1 Pro answers 12/12 at each. Flash-1017's two failures at the shortest length trace to one State error of the kind in the K3 example of Section 5: the State assigns a coordinator to the wrong production line, so Access retrieves genuine prices from that line; two State-only questions also expose the error. The original tasks thus run on a tenfold longer source with their validation checks retained, and where a failure occurs, the stage measurements place it in State rather than in Access or Answer.
W2: harder tasks expose failures that the diagnostics locate. In a separate study on two regenerated Assembly worlds, each answer depends on the records of 3, 6, or 9 components (the parts combined into a kit), while the source stays at about 21.4K tokens and the State and Access limits are fixed. Flash-1017 answers 12/12 at every setting; K3.1 Pro answers 12/12, 12/12, and 6/12. All six errors come from one State that assigns three coordinators to the wrong production lines, an error kept through Update; Access executes, and the answers follow correctly from the wrong lines' records. Two of these answers still match the reference final values while their per-component numbers differ, so a score that ignores per-component numbers counts 8/12, while complete-answer scoring counts 6/12. We do not claim that the number of components caused this error. The point is that the stage measurements trace a strong model's failures to the wrong line assignments in one State, which a task-level score partly hides.
Both studies use Flash-1017 and K3.1 Pro on two Assembly worlds and build and update one State per world, model, and setting, so we compare results only within each study; the original 96-world scores are unchanged. Together with our earlier replies on Kimi K3's Purchasing results (W2) and the 16-pair natural-text study (W3, Appendix G.2), we hope this addresses your concerns, and we would be glad to clarify anything further.
Thank you for the review, and for finding the idea clear and useful, the evaluation factual and verified, and the observation valuable that the same wrong answer can come from different causes needing different fixes. We respond to each weakness below; revisions are marked in blue in the PDF.
W2: "benchmark is too easy"
We agree that K3 is near ceiling overall: 539/576 planned tasks match completely (539/541 among tasks with an Answer output), and 576/576 with correct State and Evidence (Section 5.2; Table 2); the latter also confirms that every planned task is answerable as specified. Other configurations leave more headroom, for example 404/576 complete matches for DeepSeek max and 28/576 for Qwen3-30B-A3B (Table 12). We add three points:
Headroom and scaling. K3 is not at ceiling in every family: it matches 113/144 Purchasing tasks, whose requests follow retained mappings to item lists, bindings, and numerical records, and even with reference State its requests retrieve sufficient Evidence for 133/144 (Table 8). It fully solves the longest family (about 22K tokens), so difficulty here varies with dependency structure rather than length. The 96 worlds are the evaluated core, not a generator ceiling: new seeds add checked instances, and new families with their own dependency structures can be added under the same protocol and checks (Section 4.2).
Smaller models. The benchmark separates distinct limitations below the frontier (Appendix B.4; Table 10). With correct State supplied, Falcon-H1-7B answers 559/1,182 State-only questions but selects sufficient Evidence for only 66/576 tasks, and Qwen2.5-1.5B answers 422/1,182 but selects sufficient Evidence for none. Llama-3.1-8B instead fails mainly at the output interface (612 invalid-syntax and 379 invalid-shape recall outputs of 1,182). These call for different repairs.
Strong models. Near ceiling, the benchmark locates the remaining failures. K3's 37 are 35 tasks without an original Answer call, all answered correctly once correct inputs are supplied, and two answered nonmatches. For example, an inspected K3 State binds a task to the wrong production line; Access fetches that line's genuine prices and Answer sums them correctly, so the repair target is the binding. In contrast, a Qwen3.8-27B answer uses another service's lead time although State and Evidence contain the right offer (Section 5.4). Replacing only defective Evidence at Answer changes four selected DeepSeek tasks from 0/4 to 4/4 matches under a matched-format, thinking-disabled baseline, with Access unchanged (Table 14).
W1: "doc with 22K tokens" and "variation in length"
We agree that 22K tokens fits within current context windows. The benchmark targets a different question: whether memory prepared before the questions are known later guides source selection and evidence use. Stage-level failures, including omissions with the required knowledge in State, already appear with sources of 5,797 to 21,949 tokens across four families (Table 4; Section 5.3). In this range, K3 fully solves the longest family while missing tasks in shorter ones (Table 8).
To measure length effects directly, we plan a controlled comparison that adds source records while holding task dependencies fixed, so each stage can be tracked as length grows. The current lengths vary with task family, and this design separates the two.
W3: "real test pilot of 23 samples is thin"
The synthetic core carries the main evidence because it supplies what this diagnosis needs and natural corpora rarely provide: known retention requirements and verified support records for each answer (Section 4). We agree that the 23-question natural pilot is narrow; it tests whether one mechanism carries over. With reference State, State-guided BM25 yields 21/23 source-supported DeepSeek answers versus 4/23 for question-only retrieval with State supplied at Answer. It uses unchanged HotpotQA passages, one film-director template, six shared source groups, 557 to 909 tokens, and no Update; both methods retrieve two paragraphs, with different passages and token counts (Appendix D).
The new batch is designed for breadth across text types rather than count: the same protocol runs on Wikipedia multi-hop questions (HotpotQA, 2Wiki), scientific papers (QASPER, 5K to 8K tokens), and meeting transcripts (QMSum, 14K to 17K tokens), one preselected question per source, with papers and meetings read as whole units. With and without question-blind model summaries, all answers were correct with source support. Under the same catalog and reading budget, three summaries shortened lookup paths, one enabled a direct answer, and four pairs read equally. We will extend this batch to more sources per dataset, including section- and turn-level reading in papers and meetings, and report it separately with summary-construction costs.
Thank you again; we would be glad to answer any further questions.