Towards Smarter LLM-Powered Prediction Systems

Recent research has begun incorporating Large Language Models (LLMs) into various predictive modeling tasks. LLMs possess both a plethora of world knowledge from pre-training as well as the ability to receive and use new information in context. This makes them an attractive candidate for next event prediction (Temporal Point Process Forecasting) as they often possess significant prior knowledge on the domain of interest and allow for flexible and non-lossy incorporation of predictive features. In this work we accomplish two main goals. The first is to demonstrate that some of this world knowledge which gives LLMs an advantage over traditional models can be distilled into more lightweight 'custom' traditional models by using an LLM as a model builder. The second is to directly compare various LLM based approaches for predictive modeling in order to understand their various strengths and weaknesses. We evaluate 3 no-finetuning methods along with 2 finetuning based methods across 7 TPP datasets and provide extensive discussion on the various benefits of each approach and design considerations when building LLM-powered prediction systems.