- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CASCADE, a pipeline for generating causally interconnected fictional future events and realizing them as Wikipedia-style event and entity articles. It maintains an extracted fact database and applies consistency checks to support the generation of questions across four categories: single-event, multi-event, unanswerable, and open-ended.
Critical strengths
- A useful extension from isolated synthetic events to interconnected knowledge. CASCADE generates linked events and entity articles rather than independent fictional facts. This enables evaluation of cross-event reasoning, shared entities, and temporal dependencies over distributed evidence, providing a meaningful testing environment for knowledge acquisition.
- Broad diagnostic coverage supported by multiple validation procedures. The question taxonomy covers both successful reasoning and recognition of insufficient evidence. The real-event comparisons, retrieval ablations, scaling experiments, and human validation of open-ended grading make the benchmark informative beyond a single aggregate accuracy score.
Critical weaknesses
- Potential bias in benchmark construction and scoring. The authors remove questions based on model errors and discard rubric criteria that favor “known-weak” over “known-strong” models. It is unclear whether these procedures use separate development models, raising concerns that the benchmark may favor expected model behavior.
- Overgeneralized conclusions about parametric knowledge acquisition. The experiments mainly test raw-article CPT, and several training variants suffer severe performance collapse. Without checking different learning rates and checkpoints, these results may reflect training problems rather than a general limitation of parametric knowledge acquisition.
- Insufficient novelty over existing synthetic-event benchmarks. New News [1], NeoQA [2], and ParallelEvents [3] already evaluate knowledge acquisition or evidence-based reasoning using fictional events. CASCADE’s main extension is explicit cross-event causal connectivity and a broader question taxonomy. However, the experiments do not isolate causal interconnectedness from increased fact count and document coverage, leaving the substantive novelty and additional diagnostic value insufficiently established.
References
[1] Park et al. New News: System-2 Fine-tuning for Robust Integration of New Knowledge. 2025.
[2] Glockner et al. NeoQA: Evidence-based Question Answering with Generated News Events. 2025.
[3] Zheng et al. Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs. 2026.
Questions for authors
- Which models and predictions were used for the question-removal audit in Section 4.2? Was the benchmark frozen before final evaluation, and what fraction of questions was removed by category? Would the main rankings change on the pre-audit set?
- How were “known-strong” and “known-weak” models defined during rubric refinement? Were those systems disjoint from the final evaluated systems? How sensitive are open-ended rankings to removing the model-strength-based criterion filter?
- For CPT, did you evaluate lower learning rates, earlier checkpoints, and training/validation loss trajectories? Can you distinguish failure to acquire the new facts from loss of instruction-following ability?
- Can the scaling analysis separate increased factual volume from increased causal dependency—for example, by comparing connected and disconnected event corpora with matched fact counts and training exposure?
- Please reconcile the differing Qwen3-32B DPO results in Tables 1 and 10 and the agentic retrieval results in Table 1 and Section 6. Which retrieval pools and configurations correspond to the primary reported results?
6 comments
Potential bias in benchmark construction and scoring. The authors remove questions based on model errors and discard rubric criteria that favor “known-weak” over “known-strong” models. It is unclear whether these procedures use separate development models, raising concerns that the benchmark may favor expected model behavior.
We thank the reviewer and address each concern below.
(1) We never remove questions for being too easy or too hard. Model answers serve only to surface candidate errors in the benchmark. A question is removed only when the full source articles confirm a defect: an “unanswerable” question the articles answer, or an answerable one whose premise they contradict. These removals total 103 multi-event questions (2.1%). The other filters never use model answers: groundedness, arithmetic recomputed in code, and single-source sufficiency.
(2) Questions are generated by Claude-Sonnet-5 and GPT-4.1, neither of which is evaluated. GPT-5.2 is used only to flag benchmark errors. To test for self-preference, we had GPT-5.2 generate its own question set. Relative to other systems, it scores 3.7 points worse on those questions than on the original benchmark.
(3) For rubric generation, we included the weak-vs-strong check as a safeguard adapted from RRD, where we use GPT-5.2 as the strong model and Qwen3-8B as the weak model. It flagged only 1 of 314 candidate criteria and removed none of the 90 final ones. A blind annotator scored 480 answers from four systems, and the rubric grader agrees with these ratings at a pooled within-question Spearman of 0.66 (p < 1e-5 for every type). Neither the annotator nor the rubric rated a weaker model’s answer above a stronger one’s in any case. We will clarify these procedures in the appendix.
Which models and predictions were used for the question-removal audit in Section 4.2? Was the benchmark frozen before final evaluation, and what fraction of questions was removed by category? Would the main rankings change on the pre-audit set?
The audit details can be found in appendix D. The audit covered only questions that GPT-5.2 (BM25 full-article retrieval) answered incorrectly or abstained on. GPT-5.2 served as auditor, with the full source articles and without the original grade. It removed questions whose premise the articles contradicted or that were wrongly labelled unanswerable. It also corrected unsupported gold answers, mostly aggregative. The benchmark was then frozen, and every system is graded on the same final set. Because the audit examined only GPT-5.2’s failures, it could in principle favor that system. However, on the corrected questions, GPT-5.2 with chunk retrieval scores 8.9%. Excluding these questions leaves the ranking of all 39 systems unchanged (Spearman ρ = 0.9999). We will report the procedure and these checks in the appendix.
Overgeneralized conclusions about parametric knowledge acquisition. The experiments mainly test raw-article CPT, and several training variants suffer severe performance collapse. Without checking different learning rates and checkpoints, these results may reflect training problems rather than a general limitation of parametric knowledge acquisition.
We thank the reviewer. This concern was justified, and we ran the requested experiments.
Learning rate. We reran all four CPT recipes (Qwen3-8B and Llama-3.1-8B, full-parameter and LoRA) at a lower learning rate 1e-5 in Appendix H.2. Full-parameter CPT was limited by the learning rate. At 1e-5 it trains stably and becomes our strongest closed-book system. Single-event accuracy rises from 14.5% to 45.6% (Qwen3-8B) and from 4.5% to 36.6% (Llama-3.1-8B). Multi-event accuracy rises from 16.8% to 41.6% and from 3.8% to 33.0%. LoRA is overall worse when using a lower learning rate.
However, on single-event questions, CPT still trails retrieval with the same base model by 16–17 pp (45.6 vs. 62.5; 36.6 vs. 53.0). Additionally, the abstention rate drastically decreases, as it correctly abstains on only 19.9% and 28.9% of unanswerable questions, against 61.4% and 59.8% with retrieval. Retrieval therefore remains the better approach overall. It is more accurate on single-event questions and far more reliable at recognizing what the model does not know.
CPT. We focus on CPT because our benchmark consists of long-form text. Knowledge about these new events arrives mainly as articles, and QA pairs either do not exist or require substantial synthetic generation. CPT on articles is therefore the natural way to integrate this knowledge parametrically.
Scaling. All scaling runs use the same LoRA recipe (lr 1.5e-4, the better of the two LoRA settings in Appendix H.2). At a small scale, LoRA CPT acquires substantially more knowledge, reaching 43% on a single event. As the corpus grows, accuracy falls to 21% at 10 events, 22% at 20 and 10% at 50. Retrieval stays flat over the same range (63–71%). With the recipe held fixed, the degradation tracks the density of interconnected knowledge. Within the same event, CPT models are 10.1 pp less accurate on questions whose answer depends on an entity inherited from a causally upstream event (p < .001). Facts shared across causally linked events are where parametric storage struggles most.
Output quality. CPT and LoRA CPT outputs remain fluent and well-formed. Applying the same recipe to QA-SFT on real-event data preserves real-event performance (0.723, vs. 0.726 for the untuned base model). LoRA CPT likewise retains real-event accuracy (0.692 for Qwen3-8B; 0.511 for Llama-3.1-8B, vs. 0.535 for its base model), with under 1% degenerate outputs. The low accuracy of full-parameter CPT on Llama-3.1-8B in the original submission, despite fluent outputs, is explained by the learning rate (4.5% → 36.6% at 1e-5).
Checkpoints. We saved and evaluated a checkpoint after every epoch of CPT on 1, 10 and 20 events (full-parameter and LoRA; 44 checkpoints in total). We also evaluated 25 checkpoints across 8 QA-SFT configurations that vary the training data, the starting model (base or CPT-trained) and the adapter type. Accuracy peaks within 1–3 epochs and stays below retrieval. For Llama-3.1-8B, DPO-LoRA trained on top of CPT-LoRA stays at 20.4–24.5% across five checkpoints. We will add these results to the paper and update our findings.
Insufficient novelty over existing synthetic-event benchmarks. The experiments do not isolate causal interconnectedness from increased fact count and document coverage, leaving the substantive novelty and additional diagnostic value insufficiently established.
We thank the reviewer for this point. We have added section 7.3 and appendix I for causal structure analysis. CASCADE’s distinguishing feature is that events are causally linked, and new analyses show this connectivity affects difficulty independently of fact count, document coverage and retrieval.
Specifically, for each multi-event question, we characterize whether the required events are directly adjacent, farther apart in the graph, or connected by more directed paths. We fit linear models controlling for total and mean fact count, article length, the numbers of events and documents involved to establish the relationships between question difficulty and the connectivity characteristics. Unless noted, to measure quality difficulty, we take the average results across 23 retrieval-based systems (GPT-5.2, Gemini-2.5-Pro, Qwen3-8B and Llama-3.1-8B under BM25 RAG, IRCoT, self-query and DPO/GRPO/OPD fine-tuning).
(1) Connectivity matters beyond fact count and coverage (See Section 7.3 and Table 4). Each extra hop lowers accuracy by 2.3 pp, and a direct edge raises it by 6.6 pp (p < .001). Among questions matched on event count, fact-count decile and topic, each hop still costs 3.6 pp (p = 2e-13). Together, they show that questions demanding more hops in the causal graph are harder, while direct and alternative paths make the relevant evidence easier to connect.
To rule out that easier events simply cluster together in the graph, we keep the model unchanged and rewire the graph 300 times with degree-preserving swaps that keep it acyclic and respect decade order (See Appendix I). Each event keeps its documents, facts, degree and decade, and on average 27% of edges survive. If event-level properties drove the effect, the linear model would report the same effect for rewired graphs. Instead, the direct-edge estimate averages −2.3 pp (max +4.5), and none of the 300 reaches the real +6.6 pp (p < .0033). The real causal graph also fits accuracy better than every rewired one.
(2) Distance hurts even with full retrieval. Directly linked events share 4.2 names or entities on average, two-hop events share 0.17, and more distant events share essentially none (See Appendix I). Even when every required document is retrieved, each additional hop is associated with a 1.4-point decrease in accuracy and a 1.7-point increase in false abstention
Sensitivity to distance is a distinct capability. Per-hop losses range from 5.2 pp to near zero and are largely unrelated to overall accuracy (r = −0.29) (See Appendix I). GPT-5.2 with chunk retrieval is the most accurate system on multi-event questions (53.5%) yet loses only 1.7 pp per hop. This is a capability that only a benchmark with causally connected events can measure.
Can the scaling analysis separate increased factual volume from increased causal dependency—for example, by comparing connected and disconnected event corpora with matched fact counts and training exposure?
The 1-, 10- and 20-event corpora are nested subsets of the 50 events, with identical articles per event, so per-event fact density is fixed. What grows is total volume and the share of the causal graph in training: 3 of 79 links at 10 events (0.3 per event), 16 at 20 (0.8) and all 79 at 50 (1.6). Volume and connectivity rise together, but the largest accuracy loss coincides with the largest increase in links.
Please reconcile the differing Qwen3-32B DPO results in Tables 1 and 10 and the agentic retrieval results in Table 1 and Section 6. Which retrieval pools and configurations correspond to the primary reported results?
We apologize for the confusion. Table 1 reports our main, verified results, and we have fixed the discrepancies in the text.