- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper audits text-conditioned forecasting by distinguishing forecast response, directional alignment, control-relative benefit, and embedding-source effects. It evaluates a fixed Qwen model on 240 paired synthetic processes and compares original, permuted, and random embeddings across 16 matched GPT-2 training units. Its main conclusion is that apparent text-related effects depend on the comparator, event family, and readout.
Critical strengths
The paired event worlds and explicit text controls allow response, direction, and error to be examined separately. Including no-auxiliary-context and metadata-only comparisons helps expose conclusions that would change if incorrect text were the only baseline. Matching non-target initialization in the embedding study also improves the source comparison.
The paper carefully distinguishes original analyses from follow-ups and states the assumptions behind its statistical procedures. This transparency makes the limits of its findings easier to assess.
Critical weaknesses
The GPT-2 experiment is not specified well enough to reproduce. Appendix E explicitly omits layer-level cross-attention placement, adaptation rank, and complete numerical encoder and prediction-head definitions. It also leaves the implemented squared-loss normalization unresolved. Reconstructing statistics from preserved losses is useful, but cannot substitute for an independently testable model and evaluation procedure. These omissions affect a central part of the evidence.
The contribution supported by the remaining evidence is narrow. The main prompting experiment uses one Qwen system and three synthetic event families. The GPT-2 analysis centers on W, an integrated unsigned correct/incorrect loss gap whose ordering need not match forecasting utility. The authors correctly acknowledge this distinction, but the source effects on W offer limited guidance about useful forecasting behavior. The paper needs stronger evidence that its audit yields an empirical insight or changes evaluation conclusions beyond the particular systems and readouts examined.
The paper also does not compare against established text-conditioned forecasting methods or a standard benchmark. Without such comparisons, it is difficult to judge whether the reported comparator and readout dependence is a property of the task or of the particular synthetic setup. In addition, the provenance of the main figure is unclear; the authors should state whether generative tools were used to produce it and provide the underlying plotting code or data.
Questions for authors
- Please provide the complete GPT-2 architecture and implemented loss normalization, including variance convention and zero-scale handling. Which trained states and experimental inputs are available for independent model re-evaluation or retraining?
- Please clarify which substantive empirical conclusion requires W and is not already established by the reported forecast-error and signed-benefit comparisons. How should that conclusion inform evaluation practice?
- For additional validation, does the comparator/readout dependence persist on a fresh process cohort and a representative trained text-conditioned forecasting system?
- Which established baseline or benchmark should readers use to situate these results?