- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes full-procedure null calibration to address significance-test failures caused by feedback loops in LLM alpha mining. The method reruns the unchanged search many times on shuffled data and sets the significance threshold based on those runs.
Critical strengths
- The authors study an important problem of significance test failure in LLM alpha mining.
- The authors provide solid theoretical analysis of why feedback-driven search invalidates significance tests.
- The authors conduct multi-setting experiments to verify the impact of feedback-based search on significance testing.
Critical weaknesses
- The proposed method relies on an appropriate null-hypothesis dataset, which may be difficult to construct for real-world financial time series.
- Existing experiments are conducted using GPT, while the performance of the method across different LLMs has not yet been evaluated.
- The paper did not provide ablation studies on the granularity and frequency of the feedback provided to the LLM.
Questions for authors
- How do the authors prevent errors and hallucinations generated by LLMs from affecting the results of statistical inference?
- Could the authors evaluate additional LLMs, such as Qwen and DeepSeek, to demonstrate the robustness of the proposed method?
Response to Reviewer jXdS: the null construction, other LLMs, feedback ablations, and LLM errors
Thank you for the review. We go through each point below, and we say plainly where the paper has a limit.
W1. The method needs an appropriate null dataset. That's the method's one assumption, and we agree its scope should be stated clearly. The Discussion and the Limitations (H.8) now do that. In the setting of the paper the requirement is mild, because we never model the time series. The within-date permutation only reshuffles, on each day, which stock gets which return. Every feature's time series, each day's return distribution and the dependence among candidates that runs through the features stay exactly as they are, and for rank statistics the null is exact by exchangeability. It does destroy each stock's return autocorrelation. The forward-shift null keeps it and reproduces the ordering for the per-candidate procedures, and on the real market the i.i.d. and Newey-West statistics differ by a median factor of 1.05 (H.8). We also check the null instead of assuming it: on it, the non-adaptive control sits at nominal with uniform p-values (Table 1, KS p=0.30). And we report the constructions that failed that check, and why (Appendix E): block resampling and a circular shift leak the target into the features, so we excluded them. The real limit is that a strategy on a single time series has no cross-section to permute, so its nulls are approximate. The paper now says so.
W2 / Q2. Other LLMs such as Qwen and DeepSeek. The language-model evidence is one model,
gpt-oss-20b. We didn't have enough time in the response period to run another one. The smallest useful check, the feedback arm and its shuffled control at R=100, takes 200 full searches with the model in the loop. So we leave other models to future work, and the Limitations say plainly that the evidence is one model. We did try two larger models before submission, and H.8 explains why neither was usable. What we can tell you is which claims depend on the model.W3. Ablations on the granularity and frequency of feedback. You're right, this is missing for the language model. On that arm the feedback format is fixed. Each round the model proposes 20 formulas, then sees the 15 best so far and 10 random others, with scores. We didn't have time to vary the table size or the batch size in the response period, and the paper lists this as a limitation.
The paper has three related experiments. They aren't the ablation you ask for, but point the same way: the more the search learns from its scores, the more the tests over-reject.
Conversely, the ladder shows a score only when it beats the best so far (0.38% of queries). Under Benjamini-Hochberg that brings genetic programming back to nominal on the US panel, though it halves what the search finds (Section 5.4).
Q1. LLM errors and hallucinations. The model never computes or reports a statistic. It only writes formula strings. Each string is parsed against a fixed grammar, and if it doesn't parse it's discarded and never scored. A valid formula is scored on the data by a deterministic evaluator. So a hallucination can only be an invalid formula, which is dropped, or a valid but poor one, which is just one more candidate. Both are part of the procedure, and the same model with the same prompt is re-run on every null dataset (prompts and grammar are hashed, with no mismatch in 2,000 replicates), so whatever errors the model makes show up in the null distribution too. That's what "for any searcher" in Proposition 2 means. Appendix D reports the one such failure we found on the reported model, which we fixed before any reported run: a reasoning model's scratchpad was being parsed as proposals.
Thank you again for the time and effort you put into this review.