- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies event-driven cross-sectional ranking. A frozen LLM scores each (Truth Social post, instrument) pair as , and a dollar-neutral book trades the two tails. The authors first point out that any strictly increasing per-event correction (shift, rescaling, batch calibration) cannot change within-event order, so it does nothing for rank IC (Prop. 1). SIGNAL-TTT instead fits one offset per (event, instrument) pair. It retrieves the k nearest past events in text-embedding space from a fixed memory and takes the kernel-weighted mean residual of the frozen score against a signed rank target. It then shrinks this offset with an empirical-Bayes gate, which is basically Bühlmann credibility. On 321 development events, 39 instruments and 9 backbones, rank IC goes from 0.0030 to 0.0186 and net return improves by +2.32 bps per event.
Critical strengths
-
The evaluation protocol is unusually careful and honest for this kind of paper. Splits are chronological, every choice is made on validation, intervals use block bootstrap, comparisons are Holm-corrected and transaction costs are included. The authors also check stream-order invariance and backbone release dates. They openly report the uncomfortable results: the development window is not a true held-out test, validation and development sometimes disagree, and the book still loses money in absolute terms. This makes the numbers easier to trust than usual.
-
The method is simple, closed-form and works with black-box APIs. The motivating point that per-event corrections can't reorder instruments is clear, and the shared-offset ablation confirms it (0.0027 vs 0.0030).
Critical weaknesses
The method is simple and the evaluation is unusually honest. They report things like val/dev disagreement and the book still losing money, which I appreciate. But I’m not convinced the gain really comes from retrieval. Most of it survives when the memory is shuffled, so it looks more like a fixed per-instrument bias correction. The edge over simple retrieval baselines also isn’t significant. Details as follows:
- It is unclear how much of the gain actually comes from event-specific retrieval rather than a static per-instrument bias correction.
- With memory outcomes fully permuted within each instrument, rank IC is still 0.0149 out of 0.0186.
- The across-event permutation null gives p = 0.069.
- The paper says the per-instrument affine calibration is the only operator with positive absolute net return, yet it doesn't appear in Table 2.
- Together, these suggest a large part of the effect is "instrument a is usually mis-centered by x", which doesn't need retrieval or TTT at all. That weakens the main story.
- The gains are small and the advantage over simple retrieval baselines is not established.
- None of the retrieval comparisons survive Holm correction (p from 0.169 to 0.921). kNN-LM is close on rank IC (0.0178 vs 0.0186), and the direct Nadaraya–Watson form reaches 0.0425 on development.
- On validation, the second-half gain is only +0.0008 and net return drops by 1.38 bps.
- 6 of 9 backbones were released during or after the development window, so look-ahead is possible for most of the average.
Questions for authors
- Please add the per-instrument affine calibration to Table 2 with rank IC, net return and the paired test against SIGNAL-TTT. If it is close, what does retrieval add?
- How do the main results look when restricted to the 3 backbones released before the development window?
- Data after April 2026 should now exist. Can you report a truly held-out window that was never inspected during development?
- (Minor) There are several writing issues, e.g. "prove the work significance" in the introduction, "method discovery event driven market prediction", and "SPY500".
5 comments
Thanks for the reply
Please update the revised PDF and mark it with a different color. Thanks!
Summarize of the revisions
Thanks so much for your timely reply, We have uploaded a revised PDF. All changes are in blue, and the body is still 10 pages. Line numbers refer to the revised PDF.
- W1 (event-specific retrieval vs. static per-instrument bias).
- The Ablation paragraph (line 453) adds that permuting outcomes across instruments within each event removes the gain (p = 0.008).
- Appendix F.4 gives the full decomposition: within-instrument permutation keeps 0.0149 rank IC, and the event-specific part is about 0.0037 (p = 0.069).
- W2 (small gains; retrieval baselines). Observation ❸ already states that the differences from the retrieval operators do not survive Holm correction. It is unchanged.
- Q1 (affine in Table 2; what retrieval adds).
- Observation ❶ (line 365) adds that SIGNAL-TTT leads affine on rank IC (0.0186 vs 0.0169). When every operator trades every event, it leads by +2.88 (k = 3) and +2.06 (k = 5) bps per event, both intervals above zero.
- Line 415 adds that affine trades 76.2 of 321 events.
- Appendix F.1 (Table 16) gives Table 2 with the affine row and the traded and net-per-traded columns. Every other operator trades 189–295 events.
- Appendix F.2 (Table 17) gives the rank-only book against frozen, affine and kNN-LM.
- Q2 (3 backbones released before the development window).
- Line 433 reports +2.32 [+0.97, +3.63] bps per event and +0.0159 rank IC on these backbones, the same gain as on all 9.
- Appendix F.3 (Table 18) adds rank IC, the rank-only book and the per-backbone values: +1.70, +3.66 and +1.61 bps.
- Q3 (held-out window after April 2026). The forward window is reported in our response comment above. The conclusion (line 488) states the performance claim for the development window.
- Minor.
- Line 079: the typo is fixed ("for event-driven market prediction").
- Line 339: "SPY500" is replaced. The line now states that events are Donald Trump's Truth Social posts and instruments are S&P 500 stocks.
Hope the revision and clarify in rebuttal can solve your concerns and consider increase the rating of the paper. If you have any continue concerns please let us know.
Response to Reviewer Uasc
Thanks again for your carefully review and hopefully our rebuttal can solve your concerns. We believe with revision can make this interesting topic better. Table and line numbers refer to the submitted PDF. Brackets are 95% CIs from the paper's 5-day block bootstrap; p-values are one-sided (sign-flip for the forward endpoint). Post-unseal marks descriptive analyses computed after the forward window was scored, with the registered evaluator unchanged; they are not new endpoints.
W1. Event-specific retrieval versus static per-instrument bias.
W2. Small gains and retrieval baselines.
Q1. Affine calibration in Table 2; what retrieval adds.
Q2. The 3 backbones released before the development window. They give the same development gain as all 9 backbones, so look-ahead does not produce it. Post-unseal pooled test, same bootstrap:
Per backbone on development: +1.70, +3.66, +1.61 bps. All 9 backbones give +2.32 bps and +0.0156 rank IC.
Q3. A held-out window after April 2026. Done. 558 events, 2026-04-13 to 2026-09-21, all 9 backbones; configuration and primary endpoint hashed into a registry on 2026-09-23, before any forward outcome was scored. In this window the frozen backbones carry no signal (mean frozen rank IC 0.0375 validation, 0.0030 development, −0.0080 forward), and no operator exceeds +0.002 rank IC. SIGNAL-TTT has a higher rank IC than frozen (+0.0021, p = 0.34), every label-free TTA method and the shared offset, and its rank-only book is ahead of frozen (+0.64 [−0.53, 1.60]). The registered net endpoint is −0.31 [−0.97, +0.35] (p = 0.78); affine, which trades 109 of 558 events, is ahead of SIGNAL-TTT on the threshold book (−2.11 [−3.19, −0.97]). The forward differences against frozen are not significant. Change: the paper states its performance claims for the development window.
Minor. Fixed: line 048, line 079 and line 339 ("SPY500" → "S&P 500").