acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Knowledge Acquisition with Causally-Related Synthetic Future Events

Abstract

Large language models rely on static pre-training corpora, resulting in outdated knowledge and motivating methods for post-hoc knowledge integration. Existing approaches for evaluating knowledge updates either suffer from rapid contamination or rely on counterfactual edits that conflict with a model's existing knowledge. In this work, we introduce CASCADE, a model pipeline that generates causally interconnected fictional future events while preserving local and global consistency across a large event graph. We use CASCADE to construct a benchmark spanning a taxonomy of question reasoning types, avoiding contamination while enabling controlled, fine-grained evaluation of knowledge acquisition and reasoning. Using CASCADE, we benchmark retrieval on proprietary and open-weight models and continued pretraining (CPT) on open-weight models, finding that retrieval consistently outperforms parametric knowledge injection and that reasoning across multiple linked events is substantially harder than reasoning on a single-event. We show that reasoning-focused reinforcement learning improves accuracy under retrieval, and that retrieval methods degrade less as the number of events increases compared to CPT. These results position CASCADE as a scalable, contamination-resistant testbed for knowledge updating and multi-event reasoning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Temporal Drift, Synthetic Generation, Knowledge Insertion, Reasoning Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4XXpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CASCADE, a pipeline for generating causally interconnected fictional future events and realizing them as Wikipedia-style event and entity articles. It maintains an extracted fact database and applies consistency checks to support the generation of questions across four categories: single-event, multi-event, unanswerable, and open-ended.

Critical strengths
  1. A useful extension from isolated synthetic events to interconnected knowledge. CASCADE generates linked events and entity articles rather than independent fictional facts. This enables evaluation of cross-event reasoning, shared entities, and temporal dependencies over distributed evidence, providing a meaningful testing environment for knowledge acquisition.
  2. Broad diagnostic coverage supported by multiple validation procedures. The question taxonomy covers both successful reasoning and recognition of insufficient evidence. The real-event comparisons, retrieval ablations, scaling experiments, and human validation of open-ended grading make the benchmark informative beyond a single aggregate accuracy score.
Critical weaknesses
  1. Potential bias in benchmark construction and scoring. The authors remove questions based on model errors and discard rubric criteria that favor “known-weak” over “known-strong” models. It is unclear whether these procedures use separate development models, raising concerns that the benchmark may favor expected model behavior.
  2. Overgeneralized conclusions about parametric knowledge acquisition. The experiments mainly test raw-article CPT, and several training variants suffer severe performance collapse. Without checking different learning rates and checkpoints, these results may reflect training problems rather than a general limitation of parametric knowledge acquisition.
  3. Insufficient novelty over existing synthetic-event benchmarks. New News [1], NeoQA [2], and ParallelEvents [3] already evaluate knowledge acquisition or evidence-based reasoning using fictional events. CASCADE’s main extension is explicit cross-event causal connectivity and a broader question taxonomy. However, the experiments do not isolate causal interconnectedness from increased fact count and document coverage, leaving the substantive novelty and additional diagnostic value insufficiently established.

References

[1] Park et al. New News: System-2 Fine-tuning for Robust Integration of New Knowledge. 2025.

[2] Glockner et al. NeoQA: Evidence-based Question Answering with Generated News Events. 2025.

[3] Zheng et al. Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs. 2026.

Questions for authors
  1. Which models and predictions were used for the question-removal audit in Section 4.2? Was the benchmark frozen before final evaluation, and what fraction of questions was removed by category? Would the main rankings change on the pre-audit set?
  2. How were “known-strong” and “known-weak” models defined during rubric refinement? Were those systems disjoint from the final evaluated systems? How sensitive are open-ended rankings to removing the model-strength-based criterion filter?
  3. For CPT, did you evaluate lower learning rates, earlier checkpoints, and training/validation loss trajectories? Can you distinguish failure to acquire the new facts from loss of instruction-following ability?
  4. Can the scaling analysis separate increased factual volume from increased causal dependency—for example, by comparing connected and disconnected event corpora with matched fact counts and training exposure?
  5. Please reconcile the differing Qwen3-32B DPO results in Tables 1 and 10 and the agentic retrieval results in Table 1 and Section 6. Which retrieval pools and configurations correspond to the primary reported results?
6 comments

Insufficient novelty over existing synthetic-event benchmarks. The experiments do not isolate causal interconnectedness from increased fact count and document coverage, leaving the substantive novelty and additional diagnostic value insufficiently established.

We thank the reviewer for this point. We have added section 7.3 and appendix I for causal structure analysis. CASCADE’s distinguishing feature is that events are causally linked, and new analyses show this connectivity affects difficulty independently of fact count, document coverage and retrieval.

Specifically, for each multi-event question, we characterize whether the required events are directly adjacent, farther apart in the graph, or connected by more directed paths. We fit linear models controlling for total and mean fact count, article length, the numbers of events and documents involved to establish the relationships between question difficulty and the connectivity characteristics. Unless noted, to measure quality difficulty, we take the average results across 23 retrieval-based systems (GPT-5.2, Gemini-2.5-Pro, Qwen3-8B and Llama-3.1-8B under BM25 RAG, IRCoT, self-query and DPO/GRPO/OPD fine-tuning).

(1) Connectivity matters beyond fact count and coverage (See Section 7.3 and Table 4). Each extra hop lowers accuracy by 2.3 pp, and a direct edge raises it by 6.6 pp (p < .001). Among questions matched on event count, fact-count decile and topic, each hop still costs 3.6 pp (p = 2e-13). Together, they show that questions demanding more hops in the causal graph are harder, while direct and alternative paths make the relevant evidence easier to connect.

To rule out that easier events simply cluster together in the graph, we keep the model unchanged and rewire the graph 300 times with degree-preserving swaps that keep it acyclic and respect decade order (See Appendix I). Each event keeps its documents, facts, degree and decade, and on average 27% of edges survive. If event-level properties drove the effect, the linear model would report the same effect for rewired graphs. Instead, the direct-edge estimate averages −2.3 pp (max +4.5), and none of the 300 reaches the real +6.6 pp (p < .0033). The real causal graph also fits accuracy better than every rewired one.

(2) Distance hurts even with full retrieval. Directly linked events share 4.2 names or entities on average, two-hop events share 0.17, and more distant events share essentially none (See Appendix I). Even when every required document is retrieved, each additional hop is associated with a 1.4-point decrease in accuracy and a 1.7-point increase in false abstention

Sensitivity to distance is a distinct capability. Per-hop losses range from 5.2 pp to near zero and are largely unrelated to overall accuracy (r = −0.29) (See Appendix I). GPT-5.2 with chunk retrieval is the most accurate system on multi-event questions (53.5%) yet loses only 1.7 pp per hop. This is a capability that only a benchmark with causally connected events can measure.

Can the scaling analysis separate increased factual volume from increased causal dependency—for example, by comparing connected and disconnected event corpora with matched fact counts and training exposure?

The 1-, 10- and 20-event corpora are nested subsets of the 50 events, with identical articles per event, so per-event fact density is fixed. What grows is total volume and the share of the causal graph in training: 3 of 79 links at 10 events (0.3 per event), 16 at 20 (0.8) and all 79 at 50 (1.6). Volume and connectivity rise together, but the largest accuracy loss coincides with the largest increase in links.

Please reconcile the differing Qwen3-32B DPO results in Tables 1 and 10 and the agentic retrieval results in Table 1 and Section 6. Which retrieval pools and configurations correspond to the primary reported results?

We apologize for the confusion. Table 1 reports our main, verified results, and we have fixed the discrepancies in the text.

Potential bias in benchmark construction and scoring. The authors remove questions based on model errors and discard rubric criteria that favor “known-weak” over “known-strong” models. It is unclear whether these procedures use separate development models, raising concerns that the benchmark may favor expected model behavior.

We thank the reviewer and address each concern below.

(1) We never remove questions for being too easy or too hard. Model answers serve only to surface candidate errors in the benchmark. A question is removed only when the full source articles confirm a defect: an “unanswerable” question the articles answer, or an answerable one whose premise they contradict. These removals total 103 multi-event questions (2.1%). The other filters never use model answers: groundedness, arithmetic recomputed in code, and single-source sufficiency.

(2) Questions are generated by Claude-Sonnet-5 and GPT-4.1, neither of which is evaluated. GPT-5.2 is used only to flag benchmark errors. To test for self-preference, we had GPT-5.2 generate its own question set. Relative to other systems, it scores 3.7 points worse on those questions than on the original benchmark.

(3) For rubric generation, we included the weak-vs-strong check as a safeguard adapted from RRD, where we use GPT-5.2 as the strong model and Qwen3-8B as the weak model. It flagged only 1 of 314 candidate criteria and removed none of the 90 final ones. A blind annotator scored 480 answers from four systems, and the rubric grader agrees with these ratings at a pooled within-question Spearman of 0.66 (p < 1e-5 for every type). Neither the annotator nor the rubric rated a weaker model’s answer above a stronger one’s in any case. We will clarify these procedures in the appendix.

Which models and predictions were used for the question-removal audit in Section 4.2? Was the benchmark frozen before final evaluation, and what fraction of questions was removed by category? Would the main rankings change on the pre-audit set?

The audit details can be found in appendix D. The audit covered only questions that GPT-5.2 (BM25 full-article retrieval) answered incorrectly or abstained on. GPT-5.2 served as auditor, with the full source articles and without the original grade. It removed questions whose premise the articles contradicted or that were wrongly labelled unanswerable. It also corrected unsupported gold answers, mostly aggregative. The benchmark was then frozen, and every system is graded on the same final set. Because the audit examined only GPT-5.2’s failures, it could in principle favor that system. However, on the corrected questions, GPT-5.2 with chunk retrieval scores 8.9%. Excluding these questions leaves the ranking of all 39 systems unchanged (Spearman ρ = 0.9999). We will report the procedure and these checks in the appendix.

Overgeneralized conclusions about parametric knowledge acquisition. The experiments mainly test raw-article CPT, and several training variants suffer severe performance collapse. Without checking different learning rates and checkpoints, these results may reflect training problems rather than a general limitation of parametric knowledge acquisition.

We thank the reviewer. This concern was justified, and we ran the requested experiments.

Learning rate. We reran all four CPT recipes (Qwen3-8B and Llama-3.1-8B, full-parameter and LoRA) at a lower learning rate 1e-5 in Appendix H.2. Full-parameter CPT was limited by the learning rate. At 1e-5 it trains stably and becomes our strongest closed-book system. Single-event accuracy rises from 14.5% to 45.6% (Qwen3-8B) and from 4.5% to 36.6% (Llama-3.1-8B). Multi-event accuracy rises from 16.8% to 41.6% and from 3.8% to 33.0%. LoRA is overall worse when using a lower learning rate.

However, on single-event questions, CPT still trails retrieval with the same base model by 16–17 pp (45.6 vs. 62.5; 36.6 vs. 53.0). Additionally, the abstention rate drastically decreases, as it correctly abstains on only 19.9% and 28.9% of unanswerable questions, against 61.4% and 59.8% with retrieval. Retrieval therefore remains the better approach overall. It is more accurate on single-event questions and far more reliable at recognizing what the model does not know.

CPT. We focus on CPT because our benchmark consists of long-form text. Knowledge about these new events arrives mainly as articles, and QA pairs either do not exist or require substantial synthetic generation. CPT on articles is therefore the natural way to integrate this knowledge parametrically.

Scaling. All scaling runs use the same LoRA recipe (lr 1.5e-4, the better of the two LoRA settings in Appendix H.2). At a small scale, LoRA CPT acquires substantially more knowledge, reaching 43% on a single event. As the corpus grows, accuracy falls to 21% at 10 events, 22% at 20 and 10% at 50. Retrieval stays flat over the same range (63–71%). With the recipe held fixed, the degradation tracks the density of interconnected knowledge. Within the same event, CPT models are 10.1 pp less accurate on questions whose answer depends on an entity inherited from a causally upstream event (p < .001). Facts shared across causally linked events are where parametric storage struggles most.

Output quality. CPT and LoRA CPT outputs remain fluent and well-formed. Applying the same recipe to QA-SFT on real-event data preserves real-event performance (0.723, vs. 0.726 for the untuned base model). LoRA CPT likewise retains real-event accuracy (0.692 for Qwen3-8B; 0.511 for Llama-3.1-8B, vs. 0.535 for its base model), with under 1% degenerate outputs. The low accuracy of full-parameter CPT on Llama-3.1-8B in the original submission, despite fluent outputs, is explained by the learning rate (4.5% → 36.6% at 1e-5).

Checkpoints. We saved and evaluated a checkpoint after every epoch of CPT on 1, 10 and 20 events (full-parameter and LoRA; 44 checkpoints in total). We also evaluated 25 checkpoints across 8 QA-SFT configurations that vary the training data, the starting model (base or CPT-trained) and the adapter type. Accuracy peaks within 1–3 epochs and stays below retrieval. For Llama-3.1-8B, DPO-LoRA trained on top of CPT-LoRA stays at 20.4–24.5% across five checkpoints. We will add these results to the paper and update our findings.

Reviewer_ban8OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The main contribution is the creation of a Wikipedia style synthetic dataset consisting of causally linked fictional future events,and questions about these events with open-ended answers. They benchmark different models with retrieval augmented generation and training on the task.

Critical strengths
  1. It is useful and to the best of my knowledge novel (related datasets like NeoQA are discussed appropiately). Synthetic datasets prevent contamination, the development of an explicit event DAG is useful to benchmark multi hop/document reasoning

  2. Benchmarking of different models and retrieval/training.

3.Human validation of LLM judge for task answers.

Critical weaknesses
  1. The rubrics and llm judges for the question answering task are described and human validated but the audit process of data generation itself is insufficently documented.
  2. The shortcoming of parametric updates can be overstated since it is possible that the hyperparameters were not extensively tuned (or not reported at least) and could work better with other settings.
Questions for authors

Clarifications: Can you describe the (llm based) data auditing in more detai. i.e which model and prompts did you use at each stage, the filtering outcomes (rates, maybe some examples).

Suggestion of future work: A human audit of the corpus contradictions or incorrect labels (instead of only measuring llm judge correlation)

3 comments

The shortcoming of parametric updates can be overstated since it is possible that the hyperparameters were not extensively tuned (or not reported at least) and could work better with other settings.

We thank the reviewer. This concern was justified, and we ran the requested experiments.

Learning rate. We reran all four CPT recipes (Qwen3-8B and Llama-3.1-8B, full-parameter and LoRA) at a lower learning rate 1e-5 in Appendix H.2. Full-parameter CPT was limited by the learning rate. At 1e-5 it trains stably and becomes our strongest closed-book system. Single-event accuracy rises from 14.5% to 45.6% (Qwen3-8B) and from 4.5% to 36.6% (Llama-3.1-8B). Multi-event accuracy rises from 16.8% to 41.6% and from 3.8% to 33.0%. LoRA is overall worse when using a lower learning rate.

However, on single-event questions, CPT still trails retrieval with the same base model by 16–17 pp (45.6 vs. 62.5; 36.6 vs. 53.0). Additionally, the abstention rate drastically decreases, as it correctly abstains on only 19.9% and 28.9% of unanswerable questions, against 61.4% and 59.8% with retrieval. Retrieval therefore remains the better approach overall. It is more accurate on single-event questions and far more reliable at recognizing what the model does not know.

CPT. We focus on CPT because our benchmark consists of long-form text. Knowledge about these new events arrives mainly as articles, and QA pairs either do not exist or require substantial synthetic generation. CPT on articles is therefore the natural way to integrate this knowledge parametrically.

Scaling. All scaling runs use the same LoRA recipe (lr 1.5e-4, the better of the two LoRA settings in Appendix H.2). At a small scale, LoRA CPT acquires substantially more knowledge, reaching 43% on a single event. As the corpus grows, accuracy falls to 21% at 10 events, 22% at 20 and 10% at 50. Retrieval stays flat over the same range (63–71%). With the recipe held fixed, the degradation tracks the density of interconnected knowledge. Within the same event, CPT models are 10.1 pp less accurate on questions whose answer depends on an entity inherited from a causally upstream event (p < .001). Facts shared across causally linked events are where parametric storage struggles most.

Output quality. CPT and LoRA CPT outputs remain fluent and well-formed. Applying the same recipe to QA-SFT on real-event data preserves real-event performance (0.723, vs. 0.726 for the untuned base model). LoRA CPT likewise retains real-event accuracy (0.692 for Qwen3-8B; 0.511 for Llama-3.1-8B, vs. 0.535 for its base model), with under 1% degenerate outputs. The low accuracy of full-parameter CPT on Llama-3.1-8B in the original submission, despite fluent outputs, is explained by the learning rate (4.5% → 36.6% at 1e-5).

Checkpoints. We saved and evaluated a checkpoint after every epoch of CPT on 1, 10 and 20 events (full-parameter and LoRA; 44 checkpoints in total). We also evaluated 25 checkpoints across 8 QA-SFT configurations that vary the training data, the starting model (base or CPT-trained) and the adapter type. Accuracy peaks within 1–3 epochs and stays below retrieval. For Llama-3.1-8B, DPO-LoRA trained on top of CPT-LoRA stays at 20.4–24.5% across five checkpoints.

We will add these results to the paper and update our findings.

The rubrics and llm judges for the question answering task are described and human validated but the audit process of data generation itself is insufficently documented.

Human audits can be found in Appendix E.3 and E.4. One author annotated 125 items in a purpose-built interface, sampled with a fixed seed and shuffled.

  • Corpus contradictions (50 items). Each item is an entity in the events: every registered fact that names the entity across all 50 events, its sub-article, and up to 30 other passages in the corpus that name it. The sample covers 15 sub-articles the automated checker flagged, 15 it did not flag, and 20 entities that appear in several events, which multi-event questions depend on. The annotator did not see the checker's verdicts. A contradiction means two statements that cannot both be true; extra detail or vaguer wording does not count. Each contradiction was graded minor or major (conflicting dates, numbers, names or roles, places, or causal order) and recorded with its type, the sources involved (article, fact registry), whether it was within one event or across events, and whether it could change the answer to a benchmark question.
  • Gold labels. 40 single-event questions (5 of each type), 20 multi-event questions from the four types not covered by our question-validity audit (shared-entity, aggregative, temporal, long-chain), and 15 questions whose gold answer the automated audit rewrote. For each, the annotator saw the full fact registry (with facts cited in the gold answer highlighted), the main article and all sub-articles of every event involved. They judged whether the gold answer is correct, partially correct or incorrect according to the documents; whether it is the only defensible answer; and, if not, why.
Audit PointsResult
Entities with any contradiction8.0%
Entities with a major contradiction0%
Contradictions that could change a benchmark answer1 of 50
Checker-flagged entities with a confirmed contradiction13.3% (2/15)
Unflagged entities with a contradiction (checker misses)6.67%
Released gold answers correct (questions not touched by the repair)98.33%
… with a single defensible answer93.3%
Gold answers rewritten by the automated audit that are correct100%

The corpus is largely consistent. The few contradictions are minor and mostly between an article and the fact registry, and only one could affect an answer. The released labels are accurate, and the automated repair produced correct gold answers. We will document the audit interfaces, the samples, and the annotations in the paper.

Can you describe the (llm based) data auditing in more detail. i.e which model and prompts did you use at each stage, the filtering outcomes (rates, maybe some examples).

Exact question audit details can be found in Appendix D and Table 5. Every audit call runs at temperature 0 and returns structured JSON. The corpus checks use GPT-4.1, the pipeline's generation model. The gold-answer audit uses a stronger, separate model (GPT-5.2), so a model does not audit its own output.

Examples filter checks.

  • Date validation, "wrong": a fact dated 2042 in an event whose article ends in 2041 and never mentions such studies.
  • Consistency, medium: a sub-article said an international organization "from January 2034 onward … contributed technical expertise", but the registered facts date the organization’s founding to early 2034, with first deliverables in March–April. The released sub-article was edited to be "from March 2034 onward".
  • Gold repair: "How many years elapsed between [EVENT 1] (June 2036) and the launch of the global accord?" The gold answer said 15 years, using the start of negotiations (May 2051). The repair used the signing date (June 2053, a registered fact), resulting in an answer of 17 years.
  • Removed: a shared-entity question asked which single organisation managed a festival and also organised an earlier exposition. The facts name two different organisations, so no single answer exists.

Dear Reviewer,

Thank you again for your thoughtful feedback. We wanted to follow up on our responses above, covering CPT reruns at a lower learning rate (CPT still trails retrieval), checkpoint and output-quality analyses, and documentation of our human and LLM-based data audits.

We’d like to check whether this addresses your concerns, or if there are any remaining questions we can help clarify before the discussion period ends. We’re happy to run further experiments or provide additional analysis if that would be helpful.

Thank you for your time and consideration.

Reviewer_X7NhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CASCADE, a framework for generating benchmarks of knowledge acquisition and reasoning using fictional future events. Unlike existing benchmarks that largely consider isolated events, CASCADE constructs a shared causal event graph and generates interconnected, Wikipedia-style articles describing the events and their associated entities.

The resulting benchmark includes 50 fictional events, 926 articles, and approximately 23K questions spanning factual recall, multi-event reasoning, unanswerable questions, and open-ended responses. The authors evaluate retrieval-based approaches, continued pretraining, and reasoning-oriented post-training across several LLMs. The experiments show that multi-event reasoning is substantially harder than single-event reasoning, retrieval generally outperforms the evaluated parametric knowledge insertion methods, and additional reasoning training can improve performance over retrieved evidence. The paper also investigates knowledge retention, calibration, and abstention.

Critical strengths
  1. The paper addresses an important evaluation problem. Constructing fictional future events is a useful way to study knowledge acquisition without relying on information that models may have encountered during pretraining. Extending such events into an interconnected world is an interesting direction, particularly for evaluating how models integrate newly introduced knowledge distributed across multiple documents.

  2. The benchmark construction and empirical analysis are thoughtful. The authors put considerable effort into generating consistent articles and questions, and include automatic verification, human quality evaluation, and checks on the grading rubric. The experiments also reveal potentially useful findings, especially the difficulty of cross-event causal-chain questions and the observation that additional retrieval rounds can improve answer accuracy while substantially reducing correct abstention. These are interesting behaviors to investigate further.

Critical weaknesses

1. The distinctive contribution of causal event relationships to the benchmark is not sufficiently established.

The central methodological contribution of CASCADE is to organize fictional events through a shared causal graph, rather than generating events independently. I find this idea interesting, but the current experiments do not sufficiently isolate what the causal structure contributes to the evaluation.

The event graph is constructed using LLM-assigned causal plausibility scores, with generated rationales and narratives that follow the sampled relationships. This provides a useful way to create coherent fictional worlds. However, many of the resulting questions appear to test the retrieval and composition of facts across documents, rather than causal reasoning as a distinct capability.

For example, the paper finds that multi-event questions are substantially harder than single-event questions. However, multi-event questions also involve considerably more entities and supporting documents. It is therefore difficult to determine how much of the additional difficulty comes from causal dependencies, rather than simply requiring models to retrieve and integrate more information.

I find the particularly low accuracy on causal-chain questions, compared with long-chain multi-hop questions, potentially very interesting. But this observation also raises a question about whether the two categories have comparable information requirements, question complexity, and evaluation reliability.

One useful direction would be to construct matched questions over causal, temporal, and other relational event graphs, controlling for the number of facts, supporting documents, and reasoning steps. The existing event graph could also support a more systematic analysis of reasoning difficulty in terms of graph distance, branching structure, and alternative causal paths. Such experiments would help establish whether CASCADE measures capabilities that existing multi-hop and multi-document QA benchmarks do not adequately capture.

Related work such as SynthWorlds: Controlled Parallel Worlds for Disentangling Reasoning and Knowledge in Language Models (ICLR 2026) is relevant here, particularly its use of structurally matched worlds to separate knowledge availability from reasoning difficulty.

I do not think the paper needs to become a causal inference benchmark in the formal SCM sense. However, a clearer account of what is meant by causal reasoning in this benchmark, and what the causal graph uniquely enables, would substantially strengthen its contribution.

2. The reliability and generality of the benchmark findings need stronger support.

CASCADE generates a large collection of questions, but the fictional benchmark is derived from a single world containing 50 interconnected events. While this produces many evaluation instances, it remains unclear how stable the conclusions are across independently generated worlds, event graphs, and narrative realizations.

The paper includes useful consistency checks, independent LLM verification, human evaluation of article quality, and human validation of the open-ended grading rubric. I appreciate these efforts. Nevertheless, the human corpus evaluation covers only five events, and the correctness and intended reasoning requirements of the generated question-answer pairs rely heavily on automatic verification.

For a benchmark intended for broader reuse, I would like stronger evidence that the questions reliably measure their intended reasoning categories. This seems especially important for causal-chain, inferential, and unanswerable questions. A targeted human audit of question validity, supporting evidence, and required reasoning steps would be helpful. Evaluating several independently generated event worlds would also provide a clearer picture of robustness.

The empirical conclusions could also be more carefully separated. In particular, the comparison between retrieval and parametric knowledge acquisition involves rather different information-access settings: retrieval methods can access the source articles at inference time, whereas CPT models must answer using knowledge encoded in their parameters. This is a meaningful comparison, but the large performance gap does not by itself establish a general advantage of retrieval over parametric knowledge learning.

The authors acknowledge that their parametric baselines are primarily based on unsupervised continued pretraining, with only limited evaluation of stronger supervised alternatives. This is an important qualification. A more controlled comparison, potentially including CPT followed by retrieval or direct long-context access to the same source material, would help distinguish knowledge acquisition failures from retrieval and reasoning failures.

Related findings have been reported in Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (EMNLP 2024). I would encourage the authors to clarify what new insight CASCADE contributes beyond this established observation.

Finally, given that the benchmark aims to identify limitations of modern language models, evaluation on more recent frontier reasoning models would strengthen its relevance. It would be particularly interesting to know whether the large performance differences between causal-chain and ordinary multi-hop questions, as well as the retrieval-versus-abstention trade-off, persist in stronger models.

Overall, I consider CASCADE a promising and potentially useful benchmark. My concerns are primarily about establishing the validity and distinctive value of its causal-event construction, rather than the motivation or feasibility of the approach itself.

Questions for authors
  1. What does causal connectivity uniquely contribute to the evaluation? Can the authors distinguish the difficulty introduced by causal event relationships from that of general multi-document reasoning? A controlled comparison with structurally matched temporal or relational event graphs, or an analysis controlling for evidence size and graph distance, would be particularly informative.

  2. Why are causal-chain questions so much harder than ordinary long-chain multi-hop questions? Is there evidence that the difference reflects a genuine causal reasoning difficulty rather than differences in question construction, evidence requirements, or grading? A small set of representative examples with independently verified supporting reasoning paths would help clarify this finding.

  3. How robust are the benchmark results across independently generated worlds? The current benchmark uses a single 50-event graph. Have the authors examined variation across different graph structures or random seeds? Also, has the validity of causal-chain and unanswerable questions been independently checked by human annotators?

  4. Can the experimental findings be further separated into knowledge acquisition, retrieval, and reasoning effects? For example, how would CPT followed by RAG compare with RAG alone, or with oracle evidence? The paper already includes useful evidence-quality controls, and a more systematic comparison would help interpret the conclusions about parametric versus retrieval-based knowledge integration.

  5. Do the main capability gaps persist in stronger frontier models? In particular, I would be interested in whether the causal-chain versus multi-hop performance gap and the reduction in correct abstention under agentic retrieval remain substantial with more recent reasoning models. These results could considerably strengthen the benchmark's relevance.

4 comments

The current experiments do not sufficiently isolate what the causal structure contributes to the evaluation. It is therefore difficult to determine how much of the additional difficulty comes from causal dependencies, rather than simply requiring models to retrieve and integrate more information.

We thank the reviewer for this point. We have added section 7.3 and appendix I for causal structure analysis. CASCADE’s distinguishing feature is that events are causally linked, and new analyses show this connectivity affects difficulty independently of fact count, document coverage and retrieval.

Specifically, for each multi-event question, we characterize whether the required events are directly adjacent, farther apart in the graph, or connected by more directed paths. We fit linear models controlling for total and mean fact count, article length, the numbers of events and documents involved to establish the relationships between question difficulty and the connectivity characteristics. Unless noted, to measure quality difficulty, we take the average results across 23 retrieval-based systems (GPT-5.2, Gemini-2.5-Pro, Qwen3-8B and Llama-3.1-8B under BM25 RAG, IRCoT, self-query and DPO/GRPO/OPD fine-tuning).

(1) Connectivity matters beyond fact count and coverage (See Section 7.3 and Table 4). Each extra hop lowers accuracy by 2.3 pp, and a direct edge raises it by 6.6 pp (p < .001). Among questions matched on event count, fact-count decile and topic, each hop still costs 3.6 pp (p = 2e-13). Together, they show that questions demanding more hops in the causal graph are harder, while direct and alternative paths make the relevant evidence easier to connect.

To rule out that easier events simply cluster together in the graph, we keep the model unchanged and rewire the graph 300 times with degree-preserving swaps that keep it acyclic and respect decade order (See Appendix I). Each event keeps its documents, facts, degree and decade, and on average 27% of edges survive. If event-level properties drove the effect, the linear model would report the same effect for rewired graphs. Instead, the direct-edge estimate averages −2.3 pp (max +4.5), and none of the 300 reaches the real +6.6 pp (p < .0033). The real causal graph also fits accuracy better than every rewired one.

(2) Distance hurts even with full retrieval. Directly linked events share 4.2 names or entities on average, two-hop events share 0.17, and more distant events share essentially none (See Appendix I). Even when every required document is retrieved, each additional hop is associated with a 1.4-point decrease in accuracy and a 1.7-point increase in false abstention

Sensitivity to distance is a distinct capability. Per-hop losses range from 5.2 pp to near zero and are largely unrelated to overall accuracy (r = −0.29) (See Appendix I). GPT-5.2 with chunk retrieval is the most accurate system on multi-event questions (53.5%) yet loses only 1.7 pp per hop. This is a capability that only a benchmark with causally connected events can measure.

Many of the resulting questions appear to test the retrieval and composition of facts across documents, rather than causal reasoning as a distinct capability.

Our taxonomy covers both compositional reasoning (aggregation, comparison, multi-hop, synthesis, shared-entity) and causal reasoning (within-event causality, chains across events, and open-ended questions on significance, timing and policy). Both require reasoning on top of factual retrieval. Given the gold article, models score 89.9% on single-hop lookups but only 31.1--61.9% on aggregative, multi-hop, comparative and causal questions. Likewise, on multi-event questions, accuracy stays low wherever composition or causal tracing is needed: 36.9% for shared-entity, 6.3% for aggregative and 18.3% for causal-chain questions.

There is a question about whether the long chain multihop and causal chain have comparable information requirements, question complexity, and evaluation reliability.

The two categories draw on comparable material: 4.4 vs. 4.6 events, 293 vs. 299 facts and 77 vs. 80 documents, at a mean distance of 1.6 hops. Long-chain questions name the events, so the model constructs a multi-hop composition over a known path. Causal-chain questions name only the outcome, so the model must identify each earlier cause, which is a more direct test of causal reasoning. An outcome can have several valid causes, so we re-graded every incorrect causal-chain answer from our strongest system against the source articles for any misgrades and accepted any supported cause. Accuracy rose only 1-7 pp, confirming that models do struggle to trace causal chains.

One useful direction would be to construct matched questions over causal, temporal, and other relational event graphs, controlling for the number of facts, supporting documents, and reasoning steps.

We thank the reviewer. Matched worlds that share facts but differ in relation type are hard to build for CASCADE. Each event’s articles inherit entities and outcomes from its causes, so changing the relation type would mean regenerating all 926 documents (about 890K words and 3,099 facts), which amounts to rebuilding the benchmark. We therefore isolate causal structure within the existing world, holding facts and documents fixed:

  • Density. We train on nested subsets of 1, 10, 20 and 50 events. Each event’s articles stay identical, while causal links per event rise from 0.3 to 1.6. Accuracy on one event’s questions falls from 43.1% when it is trained alone to 10.2% in the full corpus. On the 10-event subset’s questions, doubling the facts leaves accuracy flat (21.2% vs. 21.9%). Accuracy drops to 8.9% only in the full corpus, where causal links grow fivefold.

  • Hops. With density and focal event fixed, each extra hop lowers accuracy by 2.9 pp and raises false abstention by 1.7 pp. Events with more causal parents make temporal and long-chain questions easier (+6.4 and +5.7 pp).

SynthWorlds. SynthWorlds relabels a real, static Wikidata subgraph, which has entities but no events or causes. CASCADE generates causal relations that are absent from any pretraining corpus, so we can vary them. Across 39 systems, the accuracy change per hop ranges from −5.2 to about 0 pp and is only weakly related to overall accuracy (r = −0.29). We will add SynthWorlds to related work and discuss other relation types as future work.

CASCADE generates a large collection of questions, it remains unclear how stable the conclusions are across independently generated worlds, event graphs, and narrative realizations.

We agree that using a single world is a limitation, and we will say so explicitly in Appendix A. Building a full world is costly (926 jointly verified documents, about 890K words), but we tested stability within CASCADE:

  • Event subsets. Across 1,000 random 25/25 splits of the 50 events, system rankings agree closely (median Spearman ρ = 0.99; 5th percentile ≥ 0.99, or ≥ 0.95 among retrieval systems alone) (See Appendix A). The gap between retrieval and CPT stays within 44.7–47.6 pp on single-event questions and 25.7–29.8 pp on multi-event questions. On single-event questions, every retrieval system beats every CPT system in every split.

  • Corpus size. Across nested sub-worlds of 1, 10, 20 and 50 events, retrieval stays at 63–71%, and CPT trails retrieval at all 44 checkpoints.

  • Causal graph. The rewiring test (Point 1) shows the connectivity effect only for the graph the articles were written from. The effect therefore comes from the encoded causal links, not from particular events.

The pipeline is fully automated: new causal graphs are drawn in a seeded step from LLM-assigned plausibility scores, and the generation and verification steps run unchanged on top of them. We will release the pipeline so new worlds can be generated.

For causal-chain, inferential, and unanswerable questions, a targeted human audit of question validity, supporting evidence, and required reasoning steps would be helpful.

We agree and conducted a targeted human audit of causal-chain, inferential and unanswerable questions (6 from each of the 7 unanswerable subtypes), stratified across events. These audits can be found in Appendix E.3 and E.4. An annotator read the source documents and judged each question’s clarity, the support and uniqueness of its gold answer, whether it requires the intended reasoning and multiple documents, and its number of reasoning steps. For unanswerable questions, the annotator checked that the documents leave them unanswerable, that the subtype fits and that the question is a fair test.

The audit supports all three categories. For 37.5% of causal-chain questions, the documents support a second valid chain because the outcome has several causes. This does not explain the low accuracy, since accepting any supported cause raises GPT-5.2’s accuracy only from 14.5% to 20.1%. In the revision, we will update the scores for the causal chain questions. The full audit will appear in the appendix.

It would be particularly interesting to know whether the large performance differences between causal-chain and ordinary multi-hop questions, as well as the retrieval-versus-abstention trade-off, persist in stronger models.

To test a newer frontier model, we evaluated GPT-5.6-Sol on a stratified sample. Experiments with GPT-5.6-Sol can be found in Appendix G and Tables 17 and 18. We used the same retrieval (BM25, top-10 articles, 3 rounds), prompt and graders as the paper’s runs, and compared all three models on identical questions.

GPT-5.6-Sol is the strongest model overall (+5.0 points over GPT-5.2 on multi-event and +3.4 on single-event questions), and it answers 88.4% of long-chain multi-hop questions. Yet it answers only 22.3% of causal-chain questions. Cross-event aggregative questions are the only other type that stays comparably hard (17.6%). Over-abstention also persists: GPT-5.6-Sol falsely abstains on 28.8% of answerable single-event causal, 26.4% of cross-event temporal and 16.8% of cross-event inferential questions, even though the evidence is in the corpus.

A more controlled comparison, potentially including CPT followed by retrieval or direct long-context access to the same source material, would help distinguish knowledge acquisition failures from retrieval and reasoning failures.

We thank the reviewer and ran the suggested control.

Why we compare retrieval and training. A system can store new knowledge in its weights by training on new documents, or keep it outside the model and supply it in context. We compare these two ways of encoding new knowledge under the conditions in which new knowledge actually arrives, as long-form articles. The comparison is therefore intentionally between different information-access settings.

CPT followed by retrieval. We evaluated the CPT checkpoints from Table 1 with the same BM25 retrieval as our RAG baselines (See Appendix H.3). The results show that CPT fails at storing knowledge, not at reasoning: given the same documents, LoRA CPT models match their base models on single-event questions (63.0 vs. 62.5 for Qwen3-8B; 55.3 vs. 53.0 for Llama-3.1-8B). On multi-event questions, parametric and retrieved knowledge are complementary. CPT followed by retrieval reaches 52.0% and 42.5%, which is 10–12 pp above base retrieval and above every other open-weight system we evaluated (best: 47.8% and 34.0%). However, training models results in weaker abstention on unanswerable questions (54.3% vs. 61.4%; 29.9% vs. 59.8%).

Separating retrieval, acquisition and reasoning failures for frontier models. For RAG, answering requires retrieving the relevant text, taking up the needed facts from it and reasoning over them. We classified every incorrect answer of GPT-5.2 and Gemini-2.5-Pro, using a judge shown exactly the context the model saw (See Appendix L, Table 27 and Table 28).

% of all questionsGPT-5.2 singleGemini singleGPT-5.2 multiGemini multi
Correct73.464.046.829.7
Retrieval failure (fact not in context)1.51.31.20.8
Acquisition-from-context failure (fact in context, wrong fact stated)9.19.018.121.7
Reasoning failure (facts right, chain broken)5.55.66.26.3
Incomplete answer4.47.715.922.1
Abstained5.912.211.719.1

Retrieval failures are rare, consistent with retrieval matching the gold-article condition (73.4% vs. 73.2% for GPT-5.2). The most common failure is acquisition from context. The model has the right passage but extracts the wrong fact, conflates entities or invents specifics, especially when combining facts across events.

I would encourage the authors to clarify what new insight CASCADE contributes beyond Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (EMNLP 2024).

We agree that Ovadia et al. (2024) showed retrieval generally beats fine-tuning, and we do not claim that finding. Their multiple-choice questions are about real 2023 events, each answerable from one chunk, and those events are now in training data. CASCADE uses future-dated, causally linked events with free-form questions, which shows where and why parametric acquisition fails. LoRA CPT falls from 43% to 10% as events grow from 1 to 50, while retrieval stays at 63–71%. CPT is also 10.1 pp worse on questions that depend on upstream entities (p < .001). Retrieval systems in turn struggle to synthesize facts across events, and reasoning training (DPO, OPD, GRPO) improves their accuracy, abstention and calibration. We will cite Ovadia et al. in related work.

Dear Reviewer,

Thank you again for your thoughtful feedback. We wanted to follow up on our responses above, covering new analyses isolating the contribution of causal structure (with controls for fact count, document coverage and retrieval, plus graph-rewiring and density tests), stability across event subsets, a targeted human audit of causal-chain, inferential and unanswerable questions, results with the newer GPT-5.6-Sol, and CPT-followed-by-retrieval experiments with a breakdown of failure types.

We’d like to check whether this addresses your concerns, or if there are any remaining questions we can help clarify before the discussion period ends. We’re happy to run further experiments or provide additional analysis if that would be helpful.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.