- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to use LLMs for next-event prediction in temporal point processes, where the model has to predict the type of the next event and the time until it happens. It defines three set-ups without finetuning. In S1, Claude Opus 4.8 or Qwen 3.5 32B writes a short chain of thought and then predicts directly. In S2, an agentic Claude Opus 4.8 session runs once per dataset, reads the training data and writes a stand-alone predict.py. The resulting script runs without any LLM at inference. The paper also reports a Qwen 3.5 32B version of S2. The stated goal of S2 is to distill the LLM's world knowledge into a small model. In S3, the LLM reasons at inference time and can call the S2 model as a tool. The authors compare these set-ups with LoRA finetuning of Qwen 3.5 9B, with and without CoT conditioning, on the 7 DescribeEvent datasets. They report type accuracy, macro-F1, time MAE and latency. Opus S2 beats the best of 9 published non-LLM baselines on macro-F1 and MAE, but it falls behind Opus S1 on accuracy and F1 for GitHub Repo, GitHub User and Wikipedia. The MAE of S3 stays close to that of S2, and the finetuned 9B model has the highest accuracy on 5 of 7 datasets. From the gap between S1 and S2, the authors propose the Inference Time Prior Querying (ITPQ) hypothesis, which says that compressing an LLM's world model into a static model always loses information. The paper has four further experiments. The first looks at a shift in editor count on Wikipedia, where the finetuned model drops below S1 outside its training regime. The second varies how much training data S2 gets. The third tests CoT-conditioned finetuning. The fourth is next-POI prediction on NYC Foursquare, where S2 reaches an Acc@1 of 0.280.
Critical strengths
(1) S2 gives a useful and transparent result. One S2 build per dataset produces predictors that run without an LLM. Fig. 2 puts their latency between about 1e-6 and 2e-2 s per prediction. In Table 1, these predictors beat the best of the 9 published non-LLM DescribeEvent baselines on macro-F1 and MAE on all 7 datasets. With the 3-seed means in Table 5, the F1 advantage still holds on 6 of 7 datasets, and GDELT is the exception. Appendix K adds a next-POI result under a shared scoring harness, where S2 reaches an Acc@1 of 0.280 against 0.251 for a reproduced GETNext. Appendix I describes what each build contains. On NBA, play flags based on domain knowledge add 0.06 macro-F1 and cut time MAE by 14% compared with an otherwise identical model. Appendix I.8 also reports the negative result that the event descriptions add little beyond the labels. Practitioners who need to decide whether an LLM has to sit in the inference path will find this the most useful evidence in the paper. It is also the main reason I do not recommend a clear reject.
(2) All set-ups share one protocol, and the prompts are given in full. Every set-up uses the same benchmark and the same train/test split. Appendix H gives the S1, S2 and S3 prompts word for word, and direct finetuning reuses the S1 prompt. The S3 prompt is the S1 prompt plus a tools block, so the difference between S1 and S3 measures the effect of that block. The block gives tool access and tells the model to predict first. Prior TPP work usually studies only one of these set-ups. The design is a good template for this kind of comparison.
Critical weaknesses
(1) The main comparisons have no uncertainty estimates, and several statements in the text do not match Table 1. Table 1 reports no variance. According to Appendix G, the S2 numbers in Table 1 come from a single seed, and FT is the model picked on validation out of 3 seeds. The authors' own 3-seed S2 runs, shown in the 100% columns of Tables 3 and 5, differ from Table 1 by more than several of the gaps that the text interprets. On Wikipedia, S2 accuracy is 0.358 in Table 1 but 0.393 ± 0.031 over 3 seeds, slightly above S1 at 0.389. The Wikipedia accuracy gap used to argue that S2 lags S1 therefore does not hold. On GDELT, S2 F1 is 0.160 in Table 1 but 0.127 ± 0.046 over 3 seeds, below DE-Base at 0.153. In the other direction, the 3-seed F1 means put S2 below S1 on five datasets instead of three. Whether S1 or S2 comes out ahead depends on the metric and on how seeds are aggregated, and single-seed point estimates cannot settle it. Some statements also contradict Table 1 as printed. Line 320 says S2 significantly outperforms Qwen S1, yet Qwen S1 scores higher on GDELT and Wikipedia in both accuracy and F1. Line 336 says Qwen S2 does worse than all other approaches, yet it has the highest GDELT accuracy at 0.243, with an F1 of only 0.097.
The Qwen S2 result on GDELT, with high accuracy and low F1, points to a broader problem. Table 1 has no simple baselines evaluated the same way on the test sets, such as the majority class, repeating the last type, a first-order Markov model, or a constant median time. Appendix I mentions a few baseline numbers from the build notes but does not say which split they come from. DE-Base reports no accuracy, so the accuracy lead of FT is measured only against the LLM-based approaches, and these include the S2 predictors that the LLM built. FT is never compared on accuracy with the nine published non-LLM baselines or with simple heuristics. The numbers that are available suggest such baselines matter. Appendix I.2 reports a majority-class macro-F1 of 0.06 on GDELT. If all 7 classes enter the average, a constant predictor has macro-F1 = 2p / (7(1+p)), where p is the majority-class accuracy. This puts p at roughly 0.24 to 0.29 on that split, which is in the same range as every GDELT accuracy in Table 1. The Amazon S2 build in Appendix I.7 is a scorer that combines a recency-weighted frequency term with a smoothed first-order Markov transition, and its accuracy of 0.281 is higher than Opus S1 at 0.250. Section 6 calls finetuning "the strongest method" for using LLM world knowledge while also learning from a dataset, but this depends on the metric and the dataset. FT has the best accuracy on 5 of 7 datasets and the best F1 on 3. It also has the lowest F1 on GDELT at 0.078 and the worst MAE on GDELT and Earthquake.
(2) The paper's explanations go further than its controls can support.
First, RQ1 credits the gains of S2 to distilled world knowledge, but the pipelines in Appendix I are standard ones, such as gradient boosting, LSTM or GRU, random forest, logistic regression and a Markov scorer. Appendix C says world knowledge mainly enters the S2 model through the hypothesis class and through feature engineering. Section 3 says that direct prior knowledge was mostly not distillable under S2 and places the transfer in how features are encoded. The reported feature ablations give between 0.009 and 0.06 macro-F1, plus the 14% MAE reduction on NBA. The shipped GDELT and Amazon predictors do not use the descriptions at all. For GDELT, Appendix I.2 says the distilled prior is structural. It is carried by a prior-correction exponent that favors minority classes to raise macro-F1, and by multi-window histograms with pace features. Appendix I.3 calls the 27 Hawkes scores for Earthquake the seismology prior and lists them as the top-ranked features, but no ablation removes them. None of these ablations measures the full contribution of the LLM's domain knowledge, which would also include its choice of hypothesis class. So the margin over DE-Base cannot yet be told apart from generic ML engineering and tuning aimed at macro-F1.
Second, ITPQ is stated as a necessity, since the paper says any such compression will necessarily lose information. The evidence is the gap between S1 and S2, and the size of that gap depends on the metric and on how seeds are aggregated. In Table 1, S2 trails S1 on 4 datasets in accuracy and on 3 in F1. With the 3-seed means, the counts become 2 and 5. A gap between S1 and a few particular S2 builds cannot tell an inherent ceiling apart from the limits of those builds. The paper also does not define clearly what makes a model static beyond saying it cannot reason, so it is unclear what result would count against the hypothesis.
Third, in the distribution-shift study in Table 2, the comparison between the two FT models is informative. For a fixed test regime, FT trained on that regime beats FT trained on the other one, with 0.267 against 0.242 on stable articles and 0.436 against 0.410 on contested ones. The comparison with S1 is harder to read. Table 2 does not say which backbone S1 uses. Opus 4.8 and Qwen 3.5 32B both differ in scale from the finetuned Qwen 3.5 9B, so model scale is confounded with finetuning. Appendix B still concludes that the result "localizes the OOD failure to finetuning itself, not to model scale". In the stable regime, S1 at 0.280 beats even the FT model trained in that regime at 0.267. The crossover therefore shows up only in the contested regime, where the FT model trained in that regime leads S1 by just 0.005. Table 2 reports only macro-F1, with no partition sizes and no standard deviations. S1 always reasons before it answers, so the role of reasoning in its robustness is not isolated either. Finally, Section 6 refers to "our synthetic results" in lines 477 to 480, but the paper contains no synthetic experiment.
Questions for authors
(1) Could you report the mean and standard deviation over several runs for S1, S2, S3 and FT in Table 1? FT is currently the model picked on validation out of 3 seeds. Please also give the number of test predictions per dataset for the Opus runs. Do the Wikipedia gap where S1 beats S2 and the GDELT result where S2 beats DE-Base on F1 still hold?
(2) What accuracy, macro-F1 and MAE do the majority-class, repeat-last-type, first-order Markov and constant-median-time baselines get on each test set?
(3) Could you compare S2 with a standard GBM pipeline that uses generic text features on the same JSONL files? Could you also run S2 with the dataset name, the domain blurb and the event-type names hidden? Neither control is enough alone, since the descriptions can still reveal the domain. Together they would show how much of the margin over DE-Base needs an LLM as the model builder at all.
(4) For Table 2, which backbone does S1 use, how many test sequences fall in each regime, what happens to articles with 11 to 24 editors, and what are the standard deviations across seeds? How do zero-shot Qwen 3.5 9B, an FT model trained on all the data, and S1 without the reasoning step score on each partition? What does line 419 mean by "constructed by augmenting the Wikipedia dataset"?
(5) Which experiment do the synthetic results mentioned in Section 6 refer to?
(6) The S2 agent runs with --dangerously-skip-permissions and can pip install packages. Was it cut off from the network and from the test files? Was the active variant of the S3 tools block chosen on a validation split or on test data? Where are the results of the deferral ablation mentioned in Appendix H.3?
(7) What time period do the test sequences cover, compared with the training cutoffs of Opus 4.8 and Qwen 3.5? Appendix J shows that S1 sees real player, country and user names. If the models saw these specific events during pretraining, world knowledge could look more useful than it really is.
(8) In Table 4, four FT CoT entries are single runs on a subset of 2,000 examples, while FT uses the full training set. Which model wrote the CoTs, and why does FT CoT on GitHub User drop from 0.533 to 0.365 accuracy?