- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on test-time adaptation under delayed label setting. At prediction time, the ground truth has not been realized yet, and multiple posts falling into the same time bar share one single future market target. The authors find that standard gradient-based TTA may perform worse than a frozen baseline here, because online test samples cannot supply reliable self-supervision signals. They propose R3-TTA, a retrieval-based and gradient-free TTA method. It restricts the memory bank to only mature historical samples with known ground truth, uses Bernoulli estimator to get local probability and fuses it with logits from the frozen base model. The evaluation is built on a self-collected social-media financial dataset, covering S&P 500 and crypto volatility prediction. Multiple backbones are tested, including tree models, MLP and LLMs. However, the empirical evidence is not sufficient in current submission, missing some key ablation, reproducibility materials, and there should be more TTA baselines to be compared with.
Critical strengths
The research gap detected by the paper author is critical and worth fixing, i.e., outcomes have not occurred at prediction time and posts arriving within the same time bar share a single future return in market response prediction. This finding shows a good scientific taste.
Critical weaknesses
The experiments do not show enough generalization and effectiveness. Both cases utilize the same social media source as input and the financial mechanism show high similarity. We cannot confirm whether the observed phenomenon and R3-TTA’s advantage can transfer to other more independent tasks. Baseline comparison is also not fair enough. The paper mainly compares with gradient TTA methods, while retrieval-style TTA baselines are omitted.
Questions for authors
- Add ablation to quantify how much performance drop comes from memory contamination?
- The current submission has no separate supplementary files, including both dataset and code to show reproducibility evidence.
- Argue why the core conclusion still holds on other delayed-label prediction tasks with different data generating mechanisms?
- Add more retrieval-style TTA baselines for comparison.
3 comments
summary of rebuttal revision for por9
| Comment | Change in the revised PDF | Where |
|---|---|---|
| W1: generalisation over time | App. E.3: held-out forward window, +0.0112 [+0.0043, +0.0201], Holm p = 0.003. | App. E.3 |
| W1: generalisation over assets | The abstract reads "transfers to crypto perpetuals". The intro scope reads "SPY, with crypto perpetuals as a transfer panel". App. E.3: 16/40 significant; BTC 0.6135 → 0.6386, ETH 0.5309 → 0.5710. | Abstract; §1; App. E.3 |
| W1: generalisation over architectures | App. E.1 Table 9: 7 backbones, all positive with c07. | App. E.1 |
| W1: one account | Limitation (a): events come from one account; other event streams are future work. | App. C |
| W2/Q4: retrieval baselines | App. E.1 Table 8 gives the recomputed AdaNPC fused and kNN-LM-72 rows. App. E.3 gives the paired contrasts with a search-count column, and the forward-window contrasts. | App. E.1, E.3 |
| Q1: memory contamination | App. E.7, at ρ = 0.1 / 0.25 / 0.5 / 1: the operator falls back to the frozen model, not below it. Also the out-of-fold label and newest-memory results. | App. E.7 |
| Q2: supplementary material | The Reproducibility statement lists what the repository holds and what is added: LM scripts, the crypto pipeline, the rebuttal scripts and the selection table. | Reproducibility |
| Q3: other delayed-label tasks | App. C adds a future-work sentence. It cites Chapelle 2014, Ktena et al. 2019, Grzenda et al. 2020, Joulani et al. 2013 and Csaba et al. 2024, and makes no claims outside this benchmark. | App. C |
Hope the revision and clarify in rebuttal can solve your concern and consider increase the rating of the paper.
Additional results abut how much does unreliable memory cost?
These are the means across five random-label draws at each replacement rate, using the same 525 SPY test bars and frozen GBT as in Appendix E.7.
| Memory labels randomized | AUC gain over frozen | AUC lost versus clean memory |
|---|---|---|
| 0% | +0.0199 | — |
| 10% | +0.0162 | 0.0037 |
| 25% | +0.0155 | 0.0044 |
| 50% | +0.0090 | 0.0108 |
| 100% | −0.0031 | 0.0229 |
Half-random memory loses about half the gain. At full randomization all five estimates fell below frozen, although each 95% interval included zero. Our earlier reply overstated the evidence for a fallback to frozen.
Transfer. Excluding BTC and ETH, which earlier variants had already been tested on, leaves 38 of 38 crypto series with positive gains (median +0.0108). The series share one account’s event times and are correlated. Transfer to another event source remains open.
Baselines and code. Fixed-grid kNN-LM also improves on frozen and exceeds R3-TTA on FT-Transformer by 0.0076 [0.0029, 0.0125] (E.1, E.3). A gradient-only comparison would miss this strong baseline. Tabular evaluation code is public. The promised language-model outputs, crypto pipeline and rebuttal scripts remain outstanding. Anonymous repository
Do these results address your concerns about contamination and retrieval baselines? we would be grateful for any further comments, and we would ask you to consider whether the revised evidence solved your concerns also increase the overall score.
Best
The authors
Response to Reviewer por9
Thank you for the review.
W1: generalisation beyond one source. Beyond the 2026 equity test panel:
We agree that all events come from one account, and we will state this as a limitation. The tabular backbones and the routing spaces use timing and lagged market features, not text (l.205–206). The operator does not use the account's wording, but the event times come from one account; other event streams are future work.
W2 / Q4: retrieval-style baselines. Table 2 already contains eight retrieval comparators on the same matured history (l.341; App. Table 6): T3A (NeurIPS 2021), nearest centroid, kNN vote, AdaNPC vote and AdaNPC fused (ICML 2023), Nadaraya–Watson, and kNN-LM in a selected and a fixed-grid version. While checking them we found incorrect rows in Table 2; the corrected values (GBT / MLP / FT) change the comparison:
Q1: memory contamination (new experiment). We replace a random fraction ρ of the memory labels with random 0/1 labels (GBT, 5 seeds, re-run environment where frozen = 0.7652):
Q2: supplementary material. The code and data files are at anonymous.4open.science/r/R3TTA (l.506–514). This includes the operator, the baselines, the hashed protocol, the evaluation scripts, and the post and bar files used. Before the end of the discussion period we will add the language-model scoring scripts with frozen outputs, the crypto pipeline, the rebuttal experiment scripts and the 16-candidate selection table.
Q3: other delayed-label tasks. We claim one specific result: when predictions are committed before horizon-delayed labels mature, tuning-free local smoothing over admissible memory improves discrimination, and unsupervised gradient TTA does not. The same constraint appears in other settings:
The admissibility rule, the bar-level memory and the hashed protocol carry over to these settings without change. §5 will discuss them as future work, without claiming results outside our benchmark.
We ask you to reconsider the score in light of these additions.