acceptodds
Under review as a conference paper at ICLR 2027

What Does Text Add to Forecasting? A Controlled Audit

Abstract

In our controlled studies, text-induced forecast changes do not consistently translate into lower error, and embedding-source rankings depend on the evaluation metric. We distinguish forecast response, control-relative benefit, semantic selectivity, and embedding-source effects using paired event worlds and matched training. Across 240 synthetic processes, a fixed Qwen instruction model favors correct over incorrect event descriptions on average and moves forecasts in the event's direction, but the benefit is concentrated in level changes and reverses for capacity constraints. Follow-up comparisons on the same inspected cohort have negative mean scores against no auxiliary context () and metadata only (): resampling intervals are negative, while conservative bounded intervals include zero. Across 16 matched GPT-2 training units, original and permuted pretrained embeddings both reduce an integrated unsigned loss gap relative to random embeddings, yet final error and signed-benefit readouts rank the sources differently. These findings identify concrete dependence on the comparator, event family, and readout, without establishing general semantic benefit or a shared semantic mechanism.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, text-conditioned forecasting, multimodal forecasting, language models, controlled evaluation, semantic conditioning, model attribution

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_17prOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper audits text-conditioned forecasting by distinguishing forecast response, directional alignment, control-relative benefit, and embedding-source effects. It evaluates a fixed Qwen model on 240 paired synthetic processes and compares original, permuted, and random embeddings across 16 matched GPT-2 training units. Its main conclusion is that apparent text-related effects depend on the comparator, event family, and readout.

Critical strengths

The paired event worlds and explicit text controls allow response, direction, and error to be examined separately. Including no-auxiliary-context and metadata-only comparisons helps expose conclusions that would change if incorrect text were the only baseline. Matching non-target initialization in the embedding study also improves the source comparison.

The paper carefully distinguishes original analyses from follow-ups and states the assumptions behind its statistical procedures. This transparency makes the limits of its findings easier to assess.

Critical weaknesses

The GPT-2 experiment is not specified well enough to reproduce. Appendix E explicitly omits layer-level cross-attention placement, adaptation rank, and complete numerical encoder and prediction-head definitions. It also leaves the implemented squared-loss normalization unresolved. Reconstructing statistics from preserved losses is useful, but cannot substitute for an independently testable model and evaluation procedure. These omissions affect a central part of the evidence.

The contribution supported by the remaining evidence is narrow. The main prompting experiment uses one Qwen system and three synthetic event families. The GPT-2 analysis centers on W, an integrated unsigned correct/incorrect loss gap whose ordering need not match forecasting utility. The authors correctly acknowledge this distinction, but the source effects on W offer limited guidance about useful forecasting behavior. The paper needs stronger evidence that its audit yields an empirical insight or changes evaluation conclusions beyond the particular systems and readouts examined.

The paper also does not compare against established text-conditioned forecasting methods or a standard benchmark. Without such comparisons, it is difficult to judge whether the reported comparator and readout dependence is a property of the task or of the particular synthetic setup. In addition, the provenance of the main figure is unclear; the authors should state whether generative tools were used to produce it and provide the underlying plotting code or data.

Questions for authors
  1. Please provide the complete GPT-2 architecture and implemented loss normalization, including variance convention and zero-scale handling. Which trained states and experimental inputs are available for independent model re-evaluation or retraining?
  2. Please clarify which substantive empirical conclusion requires W and is not already established by the reported forecast-error and signed-benefit comparisons. How should that conclusion inform evaluation practice?
  3. For additional validation, does the comparator/readout dependence persist on a fresh process cohort and a representative trained text-conditioned forecasting system?
  4. Which established baseline or benchmark should readers use to situate these results?
Reviewer_FHJwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how textual information influences forecasts. The paper was extremely hard to read. As best I understand, the paper has two sets of experiments.

For the data, they generate 240 synthetic time series, each with 96 historical and 24 future observations. For each series, they also generate transformations into two alternate futures across the categories “level change” (shift the whole future up or down), “timed interruption” (shift a subset of the future up or down), and “capacity constraint” (give the future values a ceiling or a floor).

In the first experiment, they study what happens when you give a model different textual descriptions of what happens in the future. They give a Qwen model the 96 historical data points and textual information, and ask it to forecast the 24 next data points. The textual information is varied across the actual transformation, the opposite of the actual transformation, neutral context, no context, and a few other textual treatments. They find that giving the actual transformation outperforms its opposite, but giving no context performs even better on average.

In the second experiment, they study what happens when you change a model’s token embeddings. They take a frozen GPT-2 model and train additional parameters on two off-the-shelf forecasting datasets, using the standard token embedding matrix, a permuted one, and a random one. They then evaluate these models on the synthetic time series along with the actual and opposite descriptions of the transformation. They study variation in a metric they define related to the difference in errors between text about the actual transformation versus its opposite.

Critical strengths

None

Critical weaknesses

These experiments are trivial and not well motivated. For example, why these choices of transformations, why the view that these specific written descriptions are representative of how these transformations are described, how are we meant to interpret the W metric.

The paper is quite underspecified. For example, I could not find the data generating process for the base synthetic time series in the paper. And in the linked repository I could only find the generated series, not the code describing what parameters and data generating process were used to generate them.

Questions for authors

How, exactly, was the base synthetic data generated?

Reviewer_ZtMtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper audits what text adds to forecasting. The paper argue that a forecast changing in response to text is not the same as the forecast getting better. Using 240 synthetic time series with paired event worlds it tests a frozen Qwen2.5-7B model prompted with correct, incorrect, neutral, and other text. The model moves forecasts in the event’s direction, and correct text beats incorrect text on average. But that gain comes almost entirely from one event type. Another study with GPT-2 was done.

Critical strengths
  1. The question is a good one. Whether a text-driven change in a forecast actually makes it better is something should be investigated properly. The paper separates “the forecast moved,” “the forecast moved the right way,” and “the forecast got more accurate”.

  2. The controlled design is clean in principle. Paired event worlds that share history, base future, and noise let the authors isolate the effect of the event itself. Using several comparators (incorrect text, no-event, no auxiliary context, metadata only) shows that the conclusion flips depending on the baseline, which is the paper’s most useful finding.

Critical weaknesses
  1. The paper doesn’t have any usable contribution and it’s very hard to read. The takeaway is something most researchers already know, and the paper offers no new method, benchmark, or fix. It’s also buried under many jargons and incoherently formed paragraphs that is very difficult to follow.

  2. The experiments are too narrow to support the claims. Everything rests on one 7B model (Qwen2.5-7B-Instruct) with one prompt format, three hand-built synthetic event types and a GPT-2 study with 16 non-independent training units. The paper’s own limitations section says it can’t generalize beyond this. The title “What does text add to forecasting?” promise far more than a single small-model setup on toy data.

  3. The core results are weak. The headline findings are mostly nulls or mixed.

Questions for authors

What is this? "Here, “review-driven” and “post-review” refer to pre-submission scrutiny of the initial results, not to official ICLR 2027 reviews."

Check the weakness section

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.