- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces NARC: puzzles made of 3–8 colour grids with one grid masked, plus a short story clue that never mentions grids. A (puzzle, model) pair has the "NARC property" when the model reconstructs the masked grid from grids + clue but fails with either alone. A diagnostic battery then perturbs the winning condition. The order test shuffles the grids three times. The form test replaces the clue with an alphabetised content-word list.
existing puzzles. On 633 puzzles and 8 open-weight models, 68.7% of the 2163 NARC cells fail every keyword version. Cross-model profiles attribute "grids suffice" to the puzzle for 2% of puzzles and to the model for 32%. A stronger closed model keeps the property on 47% of 226 human-authored puzzles, with lower form dependence.
generated puzzles. The Narrative Hierarchy Model (NHM) is a dual-surface grammar extending the Random Hierarchy Model (RHM): every event rewrites into a sentence and a grid transition. It yields 21864 TinyStories puzzles whose ambiguity and determinacy are certified with respect to the extracted event chain. It also yields a "grammar text" that renders the chain as plain sentences.
Critical strengths
- A clear unit of measurement. The NARC property is defined per (puzzle, model), and the cross-model profile (Table 4) separates puzzle defects from model limits. This is cleaner than per-puzzle labels.
- Unusually candid reporting:
- the grading-key bug and the rescoring of 2236 trials (A. B);
- verdict instability (Table 8);
- partial tuning of the core to the model pool (Table 7);
- failed pre-registered predictions (prediction iii; layer-ordered probes).
- The dual-surface grammar is a good instrument. It yields a text control faithful to the grids by construction.
- The conventions control (Table 17) separates "teaching the code" from "supplying events": conventions alone solve 37 pairs against 53 for grids alone; conventions + story solve 66 against 56 for the story.
- When the grammar text misses, 99% of its cells are still correct (Para 4.1), which supports the self-teaching check.
Critical weaknesses
- The generated benchmark does not exhibit the phenomenon in the title, and the learning study lacks the baseline that would attribute its accuracy to the text.
- The story's effect on generated puzzles is consistent with zero. Table 13 sums to 56 story+grids solves against 53 grids-only solves over 1600 model–puzzle pairs.
- Per model, the story turns a failure into a success in 47 pairs (the "NARC (story)" column). It turns a grids-only success into a failure in 44 pairs (grids-only 53, minus the 9 pairs that succeed in both).
- An exact McNemar test gives p ≈ 0.83. Only 9 of the 53 grid-solvable pairs survive adding the story.
- So the "1 to 11 NARC cells with the story" (para 4.2) are as numerous as the reverse flips. For all eight models, the story behaves like noise or a distractor. Only the grammar text (425 solves) shows a channel effect.
- The one positive story result is a single closed model on 40 puzzles, single sample: the story solves 7 of the 30 grid-unsolvable puzzles, against 18 for the grammar text (Table 12).
- Section 5 has no text-free baseline. There is no grids-only or mismatched-text 4M model at any budget (Tables 14–15).
- The label prior is strong: on the 70483-story shard, EMOTE is the masked primitive in 10740 of 15475 puzzles (69%, A. F).
- A grids-only learner may therefore reach a large share of the story's 40.7%. Without that number, claim (iv) cannot be attributed to the text.
- The best checkpoint is also selected on the same 600 held-out puzzles that are reported (A. F).
- The closing bridge does not hold. para 1 and para 7 say the battery and the learned representations "tell the same story… narratives supply the event structure that enables solutions to novel puzzles". But the two parts measure different things.
- Part I asks whether a pretrained reader uses an unseen clue.
- Part II is supervised text-to-grid learning on about 19K in-distribution puzzles under fixed conventions.
- Neither part tests solving novel problems through narrative in the ARC sense.
- The battery does not isolate narrative form; it isolates relational event content.
-
The keyword ablation removes more than form. The clue is deduplicated, alphabetised, and stripped of 120 function words, including 'after, again, all, because' (A. B). This deletes:
- argument structure (who did what to whom);
- temporal and causal connectives;
- negation;
- repetition.
In Figure 1, the decisive sentence "This made Bob happy too" leaves words like 'bob, happy, made' in a list that also contains 'ally, sad, happy'. The list cannot say whose bar rises. Failing this ablation therefore shows the reader needs compositional semantics. It does not show the reader needs a narrative.
-
The missing control is a content-matched non-narrative clue (an event list or literal instruction), and ideally also a sentence-shuffled story.
-
The paper's own evidence points the other way:
- A. D: "clinical restatements of the events matched or beat richer tellings".
- para4.1–4.2: the plain grammar text beats the story by a wide margin.
The last sentence of the abstract ("what a narrative contributes is event structure") concedes this. The title, the introduction's claim to test the "widely made claim that narrative scaffolds reasoning", and the stated hypothesis of "metaphorical transfer… via synonymy" are not tested by any experiment.
-
Form sensitivity is also reader-relative.
- The stronger model's lexical share rises from 8% to 27% (Table 10). This is equally consistent with a stronger reader reconstructing roles from keywords plus grids. The measure tracks reader weakness as much as clue structure.
- The ablations also lack a matched baseline: re-running the unchanged both-condition K = 3 times on the same cells. Only 54% of stored core NARC pairs hold across three repeats (Table 8), so part of the "strong" and "narrative" shares may be non-replication of the original success rather than an effect of the perturbation.
- The contamination bound contradicts the paper's own table, and the evaluation pool is partly selected on the evaluated models.
- Public and draft rates are not "the same". para 2.2, A. A and para 7 say never-public drafts score "at the same rate" as public puzzles and that this bounds contamination.
- Table 9 and App. D report 42/69 (61%) public against 65/157 (41%) drafts. Fisher's exact test gives p ≈ 0.009.
- Whether the gap reflects contamination or selection, the stated bound does not hold.
- The evaluated pool is partly selected on the evaluated models.
- Public puzzles passed owner approval and "the automated eight-model battery" (A. A); the pass criterion is never stated.
- The 152 agent-evolved puzzles were developed against six of the eight models.
- 20 of the 34 core puzzles come from the agent series (Table 7).
- No breakdown by provenance. Tables 1 and 4 pool human-only, model-written and agent-evolved puzzles, and public and draft puzzles, without separating them. With 404 of 633 puzzles having an LLM in the loop, the label "hand-authored benchmark" (para 2 title) is misleading.
Questions for authors
- What do grids-only and mismatched-text 4M models score at each budget, for final and middle masks, with checkpoint selection on a separate validation split?
- Near 40%: claim (iv) needs rethinking.
- Near floor: claim (iv) is supported.
- On the 2163 Part I NARC cells, how does a content-matched non-narrative clue (an event list or literal instruction) perform? How does a sentence-shuffled story perform?
- Parity: the "narrative form" reading should become "relational event content".
- A story advantage: that would be the first direct evidence for narrative form.
- How do you reconcile 61% against 41% (Table 9) with "the same rate"? What exactly does the eight-model gate require? Can you report Tables 1 and 4 by provenance and by public/draft status?
- What fraction of story-only success on Part I NARC cells survives K = 3 re-runs of the unchanged both-condition? This is the baseline the order and form verdicts need.
3 comments
Revision uploaded
We have uploaded a revision. Changes by the points in your review:
- Q1, the learning study. Section 5 is rebuilt on a separate test split of 1,141 puzzles, with no-text, mismatched-story and padded learners at every budget. The no-text learner beats the story learner at every budget, so the data ratio is withdrawn, as is the reading of the story's cost as synonymy. Table 4 reports the test puzzles where picture habits give the wrong answer, split by whether the strip was seen in training.
- Q2, narrative form. Section 2.2 and Table 2 report both controls you proposed. On the 804 slots where the grids alone fail, a plain list of the story's events solves 28.1% and the story 30.5% (difference 2.4 points, 0.8 to 4.1), keywords 8.3%. This is your parity case, and we now read the keyword test as showing a need for who-did-what, not for narrative form; the verdict is renamed "keyword-insufficient". A sentence shuffle costs 10.6 points against a placebo edit that keeps the order.
- The bridge. Unseen and seen strips are reported separately. On unseen strips no text helps a learner, so the "novel puzzles" sentence is withdrawn. The two parts are presented as two measurements of one conditional question (Section 7).
- The story null on generated puzzles is reported plainly in Section 4, with a plain list written from each story (it solves exactly what the story solves), the sentence-by-sentence ladder, a shuffle of the grammar text, and the swap test.
- Q3. "The same rate" is corrected: 61% against 41%, p = .009, and since the drafts were reachable on the site we no longer claim a contamination bound. The rule by which puzzles became public is stated exactly in Appendix E. Tables by provenance and by public or draft status are in Appendix K. Appendix D counts NARC cells on puzzles created after each reader's weights were released, which cannot be in its training data: 60% of NARC cells under either merge rule.
- Pool selection. The development of agent-evolved puzzles against six readers is disclosed in Section 2. Table 1 gives per-reader settings and outcomes. The best-case count (NARC cells that no other reader solves from the grids alone) is reported for every reader and by provenance.
- Q4. Section 2.1 reports the reruns: 305 of 418 told cells hold, and best cases hold more often (81% against 44%).
We also found, after submission, that the original runs capped reply length. We reran every capped or empty trial without the cap, report both ways of merging old and new trials side by side, and give the capped verdicts in Appendix C. Appendix B lists every change.
good result and score updated
I appreciate the substantial work on the revision. This addresses the core issues I raised earlier. The uncapped reruns are a nice addition, and the direction of the correction is supportive.
My remaining point is the writing, like: the abstract still presents the headline numbers estimated on single-point.
Overall, thank you for addressing the feedback. I am raising my score from 2 to 3.
Thank you for a very careful review. We checked each weakness against our records, and the reviewer is right on nearly all of them. Since the reviews arrived we have run new experiments aimed at each one (some still ongoing), and the revision will report them:
The four questions posed led to rechecking data and doing new tests. For each new run we wrote down beforehand how the result would decide the questions.
Q1. Grids-only (aka no-text) and mismatched-text learners. We trained both at every budget and seed, chose checkpoints on the validation split, and scored on a separate test split of 1,141 puzzles. At the time of this comment (final numbers to be reported in the revision): The no-text learner solves 38.6 to 39.2% at 1,000 puzzles and 58.4 to 60.1% with all training data, above the story learner at every budget (40.4 to 49.5% at full data). The mismatched-story learner scores the same as the story learner at full data (40.5 to 51.2%). By final and middle masks, the no-text learner solves about 63% and 46%; its gap between the two is as large as any text learner's. This is the reviewer's "near 40%" case, so claim (iv) is withdrawn as the reviewer suggests. Part of why no text does so well is that the pictures repeat: more than half of the test puzzles show a strip that also occurs in training, and most hidden events are changes of the same type (change in mood), so a learner can go far on picture habits alone. The revision therefore also reports the test puzzles where those habits give the wrong answer. There, only the learner trained on plain statements of the events beats the no-text learner; the story learner does not.
Q2. A content-matched clue and a shuffled story. We have run both. The content-matched clue cannot be run on all 2,163 cells as such, since many of those clues are not stories (coordinates, design notes). We ran it on every Part I clue that tells a story, chosen by the clue and not by any outcome; a model that is not one of the readers wrote the plain statements, which were checked before any reader saw them. We will report the numbers in the revised version. Early indications point to the reviewer's "parity" case.
Q3. The 61% and 41%, the gate, and provenance. "The same rate" is wrong and the sentence is corrected wherever it appears. The question about the eight-model gate rests on decisions we didn't make explicit and in the revised version we present per-model results. Regarding provenance: human-made puzzles are made public by hand; model-assisted puzzles were hidden in a draft form until one of our models had solved it, and public otherwise. Tables by provenance and by public or draft status will be built and will be in the revision.
Q4. K = 3 reruns of the unchanged prompt. We reran the unchanged prompt and the grids alone three times each on the 418 NARC cells whose clue is a story, for four readers: three from the paper's eight, and kimi-k3, which we added because we recently gained access to it. 305 of the 418 hold: the story solves at least two of three and the grids alone none. The revision will say which numbers rest on fresh reruns.