acceptodds
Under review as a conference paper at ICLR 2027

Only the Story Completes the Grid: Hand-Authored and Grammar-Generated Puzzles for Narrative Augmented Reasoning

Abstract

ARC-style benchmarks exclude language by design; language-annotated variants show a written description can stand in for visual examples. Neither tests the widely made claim that narrative scaffolds reasoning through novel problems. We introduce NARC, Narrative Augmented Reasoning Challenges: sequences of colour grids, one masked, with a short story as the clue. The story never mentions grids: it helps only when read figuratively, a character as a bar, a mood as its height. A puzzle has the NARC property for a model when neither channel alone yields the masked grid and both together do. We evaluate 633 puzzles on eight open-weight models with an ablation battery separating lexical, order and form dependence. In 69% of narrative-necessary cases the clue's keywords are not enough; grids sufficing is a puzzle defect for 2% and a model fact for 32%. A stronger closed model keeps the property on 47% of 226 human-authored puzzles, depending less on the story's form and more on its words. We then present the Narrative Hierarchy Model, a dual-surface grammar whose every event rewrites into a sentence and a grid transition. It generates 21,864 puzzles from TinyStories, the property certified by construction, plus a control text faithful to the grids; data and generator will be released. Stating the rendering conventions outright does not substitute for the story: what the models lack is the event chain, not the code. Models trained from scratch learn the control text from four to five times fewer puzzles than the story, and paraphrases with clean labels train like the story: the cost is wording, not labels. What a narrative contributes is event structure; what it costs a learner is the variety of the surface that carries it.

Primary area:
datasets and benchmarks
Keywords:
benchmark, abstract reasoning, ARC, narrative, ablation, hierarchical generative models, synthetic data, language and reasoning, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nCYMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NARC: puzzles made of 3–8 colour grids with one grid masked, plus a short story clue that never mentions grids. A (puzzle, model) pair has the "NARC property" when the model reconstructs the masked grid from grids + clue but fails with either alone. A diagnostic battery then perturbs the winning condition. The order test shuffles the grids three times. The form test replaces the clue with an alphabetised content-word list.

existing puzzles. On 633 puzzles and 8 open-weight models, 68.7% of the 2163 NARC cells fail every keyword version. Cross-model profiles attribute "grids suffice" to the puzzle for 2% of puzzles and to the model for 32%. A stronger closed model keeps the property on 47% of 226 human-authored puzzles, with lower form dependence.

generated puzzles. The Narrative Hierarchy Model (NHM) is a dual-surface grammar extending the Random Hierarchy Model (RHM): every event rewrites into a sentence and a grid transition. It yields 21864 TinyStories puzzles whose ambiguity and determinacy are certified with respect to the extracted event chain. It also yields a "grammar text" that renders the chain as plain sentences.

Critical strengths
  1. A clear unit of measurement. The NARC property is defined per (puzzle, model), and the cross-model profile (Table 4) separates puzzle defects from model limits. This is cleaner than per-puzzle labels.
  2. Unusually candid reporting:
    • the grading-key bug and the rescoring of 2236 trials (A. B);
    • verdict instability (Table 8);
    • partial tuning of the core to the model pool (Table 7);
    • failed pre-registered predictions (prediction iii; layer-ordered probes).
  3. The dual-surface grammar is a good instrument. It yields a text control faithful to the grids by construction.
    • The conventions control (Table 17) separates "teaching the code" from "supplying events": conventions alone solve 37 pairs against 53 for grids alone; conventions + story solve 66 against 56 for the story.
    • When the grammar text misses, 99% of its cells are still correct (Para 4.1), which supports the self-teaching check.
Critical weaknesses
  1. The generated benchmark does not exhibit the phenomenon in the title, and the learning study lacks the baseline that would attribute its accuracy to the text.
  • The story's effect on generated puzzles is consistent with zero. Table 13 sums to 56 story+grids solves against 53 grids-only solves over 1600 model–puzzle pairs.
    • Per model, the story turns a failure into a success in 47 pairs (the "NARC (story)" column). It turns a grids-only success into a failure in 44 pairs (grids-only 53, minus the 9 pairs that succeed in both).
    • An exact McNemar test gives p ≈ 0.83. Only 9 of the 53 grid-solvable pairs survive adding the story.
    • So the "1 to 11 NARC cells with the story" (para 4.2) are as numerous as the reverse flips. For all eight models, the story behaves like noise or a distractor. Only the grammar text (425 solves) shows a channel effect.
    • The one positive story result is a single closed model on 40 puzzles, single sample: the story solves 7 of the 30 grid-unsolvable puzzles, against 18 for the grammar text (Table 12).
  • Section 5 has no text-free baseline. There is no grids-only or mismatched-text 4M model at any budget (Tables 14–15).
    • The label prior is strong: on the 70483-story shard, EMOTE is the masked primitive in 10740 of 15475 puzzles (69%, A. F).
    • A grids-only learner may therefore reach a large share of the story's 40.7%. Without that number, claim (iv) cannot be attributed to the text.
    • The best checkpoint is also selected on the same 600 held-out puzzles that are reported (A. F).
  • The closing bridge does not hold. para 1 and para 7 say the battery and the learned representations "tell the same story… narratives supply the event structure that enables solutions to novel puzzles". But the two parts measure different things.
    • Part I asks whether a pretrained reader uses an unseen clue.
    • Part II is supervised text-to-grid learning on about 19K in-distribution puzzles under fixed conventions.
    • Neither part tests solving novel problems through narrative in the ARC sense.
  1. The battery does not isolate narrative form; it isolates relational event content.
  • The keyword ablation removes more than form. The clue is deduplicated, alphabetised, and stripped of 120 function words, including 'after, again, all, because' (A. B). This deletes:

    • argument structure (who did what to whom);
    • temporal and causal connectives;
    • negation;
    • repetition.

    In Figure 1, the decisive sentence "This made Bob happy too" leaves words like 'bob, happy, made' in a list that also contains 'ally, sad, happy'. The list cannot say whose bar rises. Failing this ablation therefore shows the reader needs compositional semantics. It does not show the reader needs a narrative.

  • The missing control is a content-matched non-narrative clue (an event list or literal instruction), and ideally also a sentence-shuffled story.

  • The paper's own evidence points the other way:

    • A. D: "clinical restatements of the events matched or beat richer tellings".
    • para4.1–4.2: the plain grammar text beats the story by a wide margin.

    The last sentence of the abstract ("what a narrative contributes is event structure") concedes this. The title, the introduction's claim to test the "widely made claim that narrative scaffolds reasoning", and the stated hypothesis of "metaphorical transfer… via synonymy" are not tested by any experiment.

  • Form sensitivity is also reader-relative.

    • The stronger model's lexical share rises from 8% to 27% (Table 10). This is equally consistent with a stronger reader reconstructing roles from keywords plus grids. The measure tracks reader weakness as much as clue structure.
    • The ablations also lack a matched baseline: re-running the unchanged both-condition K = 3 times on the same cells. Only 54% of stored core NARC pairs hold across three repeats (Table 8), so part of the "strong" and "narrative" shares may be non-replication of the original success rather than an effect of the perturbation.
  1. The contamination bound contradicts the paper's own table, and the evaluation pool is partly selected on the evaluated models.
  • Public and draft rates are not "the same". para 2.2, A. A and para 7 say never-public drafts score "at the same rate" as public puzzles and that this bounds contamination.
    • Table 9 and App. D report 42/69 (61%) public against 65/157 (41%) drafts. Fisher's exact test gives p ≈ 0.009.
    • Whether the gap reflects contamination or selection, the stated bound does not hold.
  • The evaluated pool is partly selected on the evaluated models.
    • Public puzzles passed owner approval and "the automated eight-model battery" (A. A); the pass criterion is never stated.
    • The 152 agent-evolved puzzles were developed against six of the eight models.
    • 20 of the 34 core puzzles come from the agent series (Table 7).
  • No breakdown by provenance. Tables 1 and 4 pool human-only, model-written and agent-evolved puzzles, and public and draft puzzles, without separating them. With 404 of 633 puzzles having an LLM in the loop, the label "hand-authored benchmark" (para 2 title) is misleading.
Questions for authors
  1. What do grids-only and mismatched-text 4M models score at each budget, for final and middle masks, with checkpoint selection on a separate validation split?
    • Near 40%: claim (iv) needs rethinking.
    • Near floor: claim (iv) is supported.
  2. On the 2163 Part I NARC cells, how does a content-matched non-narrative clue (an event list or literal instruction) perform? How does a sentence-shuffled story perform?
    • Parity: the "narrative form" reading should become "relational event content".
    • A story advantage: that would be the first direct evidence for narrative form.
  3. How do you reconcile 61% against 41% (Table 9) with "the same rate"? What exactly does the eight-model gate require? Can you report Tables 1 and 4 by provenance and by public/draft status?
  4. What fraction of story-only success on Part I NARC cells survives K = 3 re-runs of the unchanged both-condition? This is the baseline the order and form verdicts need.
3 comments

Thank you for a very careful review. We checked each weakness against our records, and the reviewer is right on nearly all of them. Since the reviews arrived we have run new experiments aimed at each one (some still ongoing), and the revision will report them:

  • Narrative form (Q2). We ran the two controls the reviewer names: plain statements of each story's events, with and without the links between them, and a sentence shuffle against an edit that keeps the order. The keyword verdict will be renamed.
  • The learning study (Q1). We trained learners with no text, with another puzzle's story, and with the event text padded to story length, and moved all scoring to a separate test split. The data ratio is withdrawn and Section 5 will be rebuilt around these baselines.
  • The bridge. We measured how many test puzzles repeat a picture strip seen in training (about half) and will report results on unseen strips separately. The "novel puzzles" sentence will be qualified, and the two parts are reported as two measurements.
  • The story null on generated puzzles. We now report it plainly, and added tests of what readers do use: removing the event text sentence by sentence, and swapping two sentences so that only their order changes the answer.
  • Contamination, the gate and provenance (Q3). We will rebuild the main tables by provenance and by public or draft status, and will describe how puzzles became public as it actually happened.
  • Pool selection and reader-relativity. The agent-evolved puzzles' development against six readers (models) cannot be undone, so we disclose it. We will add a per-reader table with settings, a reader outside the original eight, and a stricter count: NARC cells that no other reader can solve from the grids alone, which separates the quality of a puzzle from the capability of a reader (model).

The four questions posed led to rechecking data and doing new tests. For each new run we wrote down beforehand how the result would decide the questions.

Q1. Grids-only (aka no-text) and mismatched-text learners. We trained both at every budget and seed, chose checkpoints on the validation split, and scored on a separate test split of 1,141 puzzles. At the time of this comment (final numbers to be reported in the revision): The no-text learner solves 38.6 to 39.2% at 1,000 puzzles and 58.4 to 60.1% with all training data, above the story learner at every budget (40.4 to 49.5% at full data). The mismatched-story learner scores the same as the story learner at full data (40.5 to 51.2%). By final and middle masks, the no-text learner solves about 63% and 46%; its gap between the two is as large as any text learner's. This is the reviewer's "near 40%" case, so claim (iv) is withdrawn as the reviewer suggests. Part of why no text does so well is that the pictures repeat: more than half of the test puzzles show a strip that also occurs in training, and most hidden events are changes of the same type (change in mood), so a learner can go far on picture habits alone. The revision therefore also reports the test puzzles where those habits give the wrong answer. There, only the learner trained on plain statements of the events beats the no-text learner; the story learner does not.

Q2. A content-matched clue and a shuffled story. We have run both. The content-matched clue cannot be run on all 2,163 cells as such, since many of those clues are not stories (coordinates, design notes). We ran it on every Part I clue that tells a story, chosen by the clue and not by any outcome; a model that is not one of the readers wrote the plain statements, which were checked before any reader saw them. We will report the numbers in the revised version. Early indications point to the reviewer's "parity" case.

Q3. The 61% and 41%, the gate, and provenance. "The same rate" is wrong and the sentence is corrected wherever it appears. The question about the eight-model gate rests on decisions we didn't make explicit and in the revised version we present per-model results. Regarding provenance: human-made puzzles are made public by hand; model-assisted puzzles were hidden in a draft form until one of our models had solved it, and public otherwise. Tables by provenance and by public or draft status will be built and will be in the revision.

Q4. K = 3 reruns of the unchanged prompt. We reran the unchanged prompt and the grids alone three times each on the 418 NARC cells whose clue is a story, for four readers: three from the paper's eight, and kimi-k3, which we added because we recently gained access to it. 305 of the 418 hold: the story solves at least two of three and the grids alone none. The revision will say which numbers rest on fresh reruns.

AuthorsOpenReview ↗

Revision uploaded

We have uploaded a revision. Changes by the points in your review:

  • Q1, the learning study. Section 5 is rebuilt on a separate test split of 1,141 puzzles, with no-text, mismatched-story and padded learners at every budget. The no-text learner beats the story learner at every budget, so the data ratio is withdrawn, as is the reading of the story's cost as synonymy. Table 4 reports the test puzzles where picture habits give the wrong answer, split by whether the strip was seen in training.
  • Q2, narrative form. Section 2.2 and Table 2 report both controls you proposed. On the 804 slots where the grids alone fail, a plain list of the story's events solves 28.1% and the story 30.5% (difference 2.4 points, 0.8 to 4.1), keywords 8.3%. This is your parity case, and we now read the keyword test as showing a need for who-did-what, not for narrative form; the verdict is renamed "keyword-insufficient". A sentence shuffle costs 10.6 points against a placebo edit that keeps the order.
  • The bridge. Unseen and seen strips are reported separately. On unseen strips no text helps a learner, so the "novel puzzles" sentence is withdrawn. The two parts are presented as two measurements of one conditional question (Section 7).
  • The story null on generated puzzles is reported plainly in Section 4, with a plain list written from each story (it solves exactly what the story solves), the sentence-by-sentence ladder, a shuffle of the grammar text, and the swap test.
  • Q3. "The same rate" is corrected: 61% against 41%, p = .009, and since the drafts were reachable on the site we no longer claim a contamination bound. The rule by which puzzles became public is stated exactly in Appendix E. Tables by provenance and by public or draft status are in Appendix K. Appendix D counts NARC cells on puzzles created after each reader's weights were released, which cannot be in its training data: 60% of NARC cells under either merge rule.
  • Pool selection. The development of agent-evolved puzzles against six readers is disclosed in Section 2. Table 1 gives per-reader settings and outcomes. The best-case count (NARC cells that no other reader solves from the grids alone) is reported for every reader and by provenance.
  • Q4. Section 2.1 reports the reruns: 305 of 418 told cells hold, and best cases hold more often (81% against 44%).

We also found, after submission, that the original runs capped reply length. We reran every capped or empty trial without the cap, report both ways of merging old and new trials side by side, and give the capped verdicts in Appendix C. Appendix B lists every change.

Reviewer_nCYMOpenReview ↗

good result and score updated

I appreciate the substantial work on the revision. This addresses the core issues I raised earlier. The uncapped reruns are a nice addition, and the direction of the correction is supportive.

My remaining point is the writing, like: the abstract still presents the headline numbers estimated on single-point.

Overall, thank you for addressing the feedback. I am raising my score from 2 to 3.

Reviewer_pKVSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors proposed NARC (Narrative Augmented Reasoning Challenges), a benchmark testing whether narrative genuinely scaffolds reasoning on novel problems, a claim neither ARC nor its language-annotated variants address. Each puzzle is a sequence of colour grids with one masked and a short story as the clue; the story never mentions grids and helps only when read figuratively. A puzzle has the “NARC property” for a model when neither the grids nor the story alone suffices but both together do. Evaluating 633 puzzles across eight open-weight models with ablations separating lexical, order, and form dependence, they find that in 69% of narrative-necessary cases keywords alone are insufficient, while a stronger closed model retains the property on 47% of 226 human-authored puzzles. They further introduce the Narrative Hierarchy Model, a dual-surface grammar generating 21,864 puzzles from TinyStories with the property guaranteed by construction. Training experiments show narrative contributes event structure, but its surface variety makes it roughly four to five times costlier to learn than a grid-faithful control text.

Critical strengths
  • The setup and task is interesting. The author extends ARC benchmarks, which are proved to be difficult for LLMs, to multimodal, forcing the model to reason across multiple modalities to answer the question.

  • The dual-surface generator is interesting, and using the same model to generate all modalities allows the generated samples to have seamless multi-modality interactions, which makes benchmark interesting.

Critical weaknesses
  • The main issue I think is the main text seems to be AI generated largely, with walls of texts that are very hard to understand, giving little to no information, while lacking analysis that's needed for a complete paper. The author stacks private terminology stacked before it’s motivated (cells, profiles, tiers, strong/partial/weak crossed with narrative/partial/lexical). Paper also includes sentences like “the finding this paper carries forward is that it’s the clue’s (latent) narrative form that explains the solving ability” that assert a conclusion the preceding paragraph doesn’t establish. These issues makes the paper very hard to review under the current draft.

  • The experiments and analysis of results are very limited in the main text. From the main text, I do not see any experiments noting the comparisons across model names, reasoning settings, etc. These are very basic analysis that a dataset paper should include. In addition, there should also be analysis on how the accuracy changes as the length of story, size of canvas, number of images, etc. changes. These are also interesting analysis that I think the authors should include.

Questions for authors

Please see weaknesses.

3 comments

We are grateful for the reviewer's patience in reading the text and have resolved ourselves to doing a full overhaul on the prose and significantly trimming AI generated text. We plan to address the weakness as follows:

  1. The revised version will be re-written with our own (human!) prose. Private terminology will be motivated and explained with brief worked examples; a small table will explain the shorthand labels we use. In addition, we agree about the sentence quoted and it will be withdrawn as claimed. In the revision we will be clearer about differences between what this project aspires towards versus what our evidence supports (including additional experiments we have run to address other reviewers feedback).

  2. We agree that the main text should have made clearer the comparisons that we did across models and their settings. Some of these were buried in Appendix C. In the revised paper we will provide these comparisons in the main text, updated with information from additional tests underway. While we had hoped to do so, we unfortunately are unlikely to be able to include variations across reasoning efforts in time for the revised version, and if not we will say so as a limitation.

  3. We have ongoing experiments about, e.g. the impact of story length on accuracy. These are nearing completion. We also have existing data about accuracy against number of grids, canvas size, clue length, and each separately for puzzles made by people, a model, and an agent (with a human in the loop). We will include analyses of those results in the revision.

AuthorsOpenReview ↗

Revision uploaded

We have uploaded a revision. Changes by the points in your review:

  1. The text is rewritten. Shorthand labels are explained where they first appear and collected in a table (Appendix A). The sentence you quoted is withdrawn, and Section 7 now separates what the evidence supports from what the project aims at.
  2. Model comparisons and settings are in the main text (Table 1): reasoning effort, reply limit, how often replies were cut off, NARC rate and best cases per reader. We did not vary reasoning effort, and say so as a limitation.
  3. Story length: a learner trained on the plain event text padded to the story's length trains as badly as the story learner (Section 5), so length and irrelevant content account for most of the story's cost. Accuracy against the number of grids, canvas size and clue length, by who made the puzzle, is in Figure 13 (Appendix K).

Appendix B lists every change from the submitted version.

Reviewer_pKVSOpenReview ↗

Thanks for the revision and updated score

Hi Authors,

Thanks for updating the manuscript. I read it and still found it to be a bit hard to understand with some AI written texts. With that said, I appreciate the effort to clean up the texts and add requested experiments. Most of my concerns are addressed, and I think the work overall is an interesting contribution to the community. Therefore, I'm raising my score from 2 to 3.

Reviewer_i4qqOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a task in which models use a story and a sequence of visible gridsto reconstruct a missing grid. The authors evaluate existing language models, generate a larger training dataset, and train small models from scratchto compare different text formats. Directdescriptions of events are easier to learn from than stories: the 4M model achieves 66.0% accuracy with direct descriptions versus 40.7%with stories.

Critical strengths
  1. The dataset supports useful controlled comparisons. The sameevents can be expressed as direct descriptions or stories while keeping the grids and answers unchanged. This provides a way to study whether models struggle to extract events fromtext or to translate those events into changes in the grid.
  2. The effect oftext format is substantial. Beyond comparing original stories with direct descriptions, the authors rewrite the descriptionsinto story-like textand again observe lower accuracy. This provides evidence that how the same events are expressed can substantially affect learning.
Critical weaknesses
  1. The need for this new task is unclear. Existing work already studies tracking entities and state changes from text, as well as using language to solve grid tasks. The paper combines these settings but does not clearly establish what new research questionor model limitation this combination reveals.
  2. The experiments do not establish a distinct role for narrative structure. The task placesinformation missing from the grids into the story, soimprovements from adding the story primarily demonstrate information complementarity. Replacing the story with keywords also removes essential relationships, such as who did what to whom. Failure underthis condition does not establish that narrative structure itself is necessary.
  3. The explanation for the higher learning cost remains incomplete. Stories differ from direct descriptions in length, sentence structure, pronoun use, and irrelevant content. The experiments do not separate these factors, so attributing the additional data requirements mainly to having multiple ways of expressing the same event is not sufficiently supported.
Questions for authors
  • What makes NARC a useful research task beyond existing state-tracking and language-guided grid benchmarks?
3 comments

Thank you for the review. The three weaknesses identified have been informative and our plan to respond to them in the revision are as follows (and we believe #1 also address the "Questions for Authors" from the reviewer).

  1. We agree we should be more clear about why this new task is needed, and why we barely engaged with the state change and entity tracking literatures in the first version. Here is how we plan to address this critique in the revision. For context, our research questions come from narrative science, which studies how narratives help scientists and people generally make sense of puzzling things. The precise sense of "narrative" is something that we will better articulate in the revision; it is not merely a description of a sequence of events because they also involve a capacity for applying stories metaphorically or figuratively. However, narrative science suffers from methodological problems and argues mostly from case studies. To know whether a narrative made the difference for one solver (human or model) on one puzzle, you must see that solver with the story and without it, from the same starting point. But a person cannot un-read a story. By contrast, a language model can be run again from the same state, and we can perform various interventions and ablations that are difficult or impossible to do with humans. That said, we do not know in advance whether a model has developed an emergent capacity for reading stories to solve problems, particularly when those stories need to be applied metaphorically. So models and puzzles have to be assessed together first: which reader and puzzle pairs repeat, and which survive simpler explanations by, e.g. using keywords. The NARC property and its follow-up tests (among others we have added) do that job. What we believe the machine learning community gains is a new diagnostic benchmark that identifies combinations of models and puzzles that can be used as instruments for transporting narrative science into the domain of machines.

  2. We agree with the reviewer's critique of the paper as written because we failed to be sufficiently nuanced in expressing how we are thinking about "narrative structure" and how that corresponds to the puzzles and tests we perform. And we agree that failing on the keywords test is too crude. We have existing data from control tests that would have helped paint a more nuanced pictured, and we are running additional controls that will be complete in the next few days. Specifically, for every puzzle from Part I we wrote a version of the story into plain statements of the events in the grids, with and without pronouns and demonstratives that tie the events together as a narrative. In short, the revision will improve the "narrative form" reading both conceptually and empirically.

  3. In regards to the points about learning costs in Part II of the paper, we have added two additional controls. First, we have added a baseline "no-text learner" that is trained with no text at all (just the grid sequences). Preliminary results are showing that the no-text baseline outperforms all the others (to our surprise), with two important exceptions. One is when all the training puzzles are used, in which case the original plain event descriptions (what we called the "grammar-text" generated by the production rules) does slightly better. The second is when we condition on those puzzles with endings that deviate from conventional ones. That is, the strips of grid sequences the grammar produces have typical types of endings that occur roughly 75% of the time. The no-text learner picks up on these trends and then, unsurprisingly, fails to solve those puzzles where the ending is different. The grammar-text learner, however, outperforms all others on these puzzles with "unconventional" endings, which suggests that the narrative is doing real work with the observable grids to steer inference towards the masked grid even when it is rarely seen in training (if at all).

We have also added a "padded learner" that is trained on the plain event descriptions with additional padding to match the length of the original children story. The padding comes from sentences from other stories, which changes length and irrelevant content while keeping the wording from the "grammar-text" version we already have. Preliminary results are suggesting that this padded learner performs as poorly as the "story learner" (the story text from the tinyStory dataset), suggesting that length and irrelevant content could account for the gap in performance. Our revision will report these findings.

AuthorsOpenReview ↗

Revision uploaded

We have uploaded a revision. Changes by the points in your review:

  1. Why the task. The introduction now states the purpose from narrative science: a model can be run with and without a story from the same state, which a person cannot, so models are candidate instruments, but only once we know which reader-puzzle pairs actually use the story. Related work (Section 6) now opens with entity and state tracking (bAbI, ProPara, OpenPI, TRIP, PIGLeT, entity tracking in language models, among others) and work comparing told and stated versions of the same content.
  2. Narrative. We state the sense we use, events in order, and distinguish a story that tells its events from a list that states them; applying the clue to the grids is the figurative step. The plain-statement controls are reported in Section 2.2: a plain list of the story's events does nearly as well as the story, and the keywords far worse. So readers need the relations between events, not the telling.
  3. Learning. The no-text and padded learners are reported in Section 5. We need to correct one thing from our first comment. The plain event text's lead where picture habits give the wrong answer comes from strips seen in training with another ending (+40 to +45 points over no text); on strips never seen in training it adds 1 to 3 points, which we cannot tell from zero. So the claim we make is narrower than "even when it is rarely seen in training (if at all)": a text that states the events helps a learner override a habit on a familiar strip. Also, "endings that occur roughly 75% of the time" should have read: a mood change is the hidden event in 68.5% of test puzzles, and the no-text learner solves 75% of those.

Appendix B lists every change from the submitted version.

Thank you for providing further clarification. Although the response is helpful, several of my key concerns remain insufficiently addressed. I will slightly revise my evaluation upward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.