- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper defines a factor as an investment question together with a fixed execution protocol. An agent reads the records available on the scoring date, may write code, and returns a score from 0 to 5 along with supporting evidence, counter-evidence, and missing information. A second loop rewrites questions from development-period returns and company cases. Candidates from every generation stay in a bank, and the set used at test time is chosen on a later validation window, then frozen. The empirical work has two parts. One asks whether the score measures the property named in the question: agreement with classic formulas, response to edited evidence, and stability across repeated runs. The other asks whether validation-selected ensembles predict returns better than the starting questions. On CSI 300 and Nasdaq-100, in both fundamental and technical settings, the selected ensemble improves IC and RankIC over the initial ensemble in three settings out of four. The test window is January to March 2026. The authors describe this as a feasibility result.
Critical strengths
1.The paper is clear about what a score is supposed to mean. Measurement and return prediction are set up as separate tests, and the write-up does not treat a high score as a claim about future profits. The formula comparison, the controlled evidence edits, and the repeated runs each answer a different question. The Amihud illiquidity question, where the agent ranks stocks in the opposite direction from the formula, is reported in the figure rather than dropped.
2.The execution protocol is concrete. The rubric, the tools, the token accounting, and the Zscaler trace make it possible to see the agent retrieve filings, compute a quantity, notice a sign error, and correct it before scoring. The case studies also keep the counter-evidence in the answer when the score is at the top of the scale.
3.The selection rule is more careful than keeping the question with the best in-sample IC. A question has to look acceptable on its weaker horizon, and the chosen set is fixed before the test months. The authors also state the setting that goes the wrong way: on Nasdaq-100 technicals, the selected ensemble is worse than the initial one on IC, RankIC, and spread.
Critical weaknesses
1.The out-of-sample results are not separable from noise. The test contains only three scoring dates, January through March 2026. The main table reports neither standard errors nor a test against the initial question set. Fundamental evaluation also uses 90-day returns, so labels from adjacent months overlap heavily. The Nasdaq-100 fundamental IC rises from near zero to about 0.10, the largest gain in the paper and the one most dependent on this short sample. The appendix reruns show that, with the question text held fixed, median validation fitness already moves by about 0.04. The validation gain of the leading Nasdaq-100 fundamental questions is about 0.04 as well.
2.The paper’s own comparison is unstable, and the gains do not clear a simple baseline. Relative to the initial question set, IC and RankIC improve in three of the four settings, but the CSI 300 long–short spread is essentially unchanged or lower, and the Nasdaq-100 technical ensemble is worse on IC, RankIC, and spread. The results may show only that selection sometimes rewrites the authors’ own initial questions. They do not show that the scores form a distinctive predictive signal.
3.Cutting off the files at the scoring date does not rule out information already present in the model weights. The executor is a general-purpose 2026 model, and the sensitivity runs use models from the same period. Retrieval is restricted to records available by the scoring date. The paper reports no knowledge cutoff, and it provides no control with an earlier-cutoff model or a retrieval-only setup.
4.The measurement experiments do not test the factors that enter the final evaluation. Formula agreement, controlled evidence changes, and repeated execution are run on classic questions or on the initial questions. The evolution instructions ask for questions on which a higher score means a higher expected return, and they allow a stably negative question to be reversed outright. After such a reversal, the score tracks the rewritten return direction rather than the original investment judgment. Measurement validity and return prediction are therefore results about two different sets of factors, and the paper does not connect them.
Questions for authors
1.On each validation and test date, how many stocks receive a score? Is that the full index, or the development subsample of 300 agent tasks spread over the year? If the validation RankIC is computed on a thin cross-section, the selection rule is fitting a noisy statistic.
2.How many questions are retained, and are they rescored on the test dates under the frozen protocol? Please list the selected questions, including any whose meaning was reversed.
3.What is the pretraining cutoff of DeepSeek-v4-flash, and were the document summaries produced by a model that had already seen text from after the scoring date? The informative check is to rescore the test panel with a model whose cutoff precedes April 2024, or with retrieval only.
4.For the evolved questions that enter the test, is there a measurement check comparable to the formula agreement figure? In particular, after a question is reversed so that its RankIC changes sign, what property does the score still measure?
3 comments
Thank you for the authors' response. My main concerns are addressed, And I think the experiment and ablation are enough to support the papers declare. However, there are fewer factor formulas released by the authors, so it is less likely to check the real expression to understand the economic logic. However I think the work is well organized and expressed and I am willing to update my score, good job.
Thank you very much for your thoughtful follow-up and your willingness to update your score. We sincerely appreciate the time and care you have put into reviewing our work. Your feedback has helped us strengthen the experimental evidence and clarify the scope of our claims.
We also appreciate your suggestion regarding factor interpretability. Making the economic intuition behind each factor directly accessible is indeed one of the motivations for our natural-language formulation. For reference, the full text of all 48 selected questions is provided in Appendix E.3 (Table 11).
Thank you again for your constructive feedback and encouraging assessment.
Thank you for the careful reading and these comments. A factor here is the investment question plus a fixed execution protocol: the agent reads company materials and returns a score with checkable evidence, so factors can express ideas requiring several sources. Measurement consistency (formula agreement 0.821/0.728; strict P>B>N in 33/40; ICC 0.862/0.704) is tested independently of returns, and the return and evolution experiments ask whether such factors plug into a factor research workflow; the revision softens the return claims, as agent subjectivity, market complexity and low signal-to-noise make consistent returns difficult. We added a sampling-variation analysis, the selected-question list, the semantic-reversal explanation, the cutoffs and a retrieval-only baseline.
W1 & Q1. Short test period and overlapping labels; samples per date
Overlap between neighboring forward-return windows is ordinary in sequence experiments: rolling-origin evaluation advances the forecast origin step by step and reuses most of the preceding history as input, and sliding windows feed the same observations into many training examples, without the shared part ever including the outcome being predicted. Our metrics are computed within a single scoring cross-section, and the gaps between development, validation and test are wide enough that no outcome window straddles a stage boundary, so overlap cannot carry information from one stage into another.
Each scoring date has a few dozen samples; we use stratified sampling, fixed panels and per-date median RankIC, taking the weaker of the two horizons. As small cross-sections still fluctuate, we resample the 300 agent tasks 5,000 times, stratified by date and industry. Only NDX fundamental is clearly above sampling variation: its IC and RankIC differences (0.096, 0.082) are about twice the bootstrap SEs (0.026, 0.030); elsewhere they are below or near the SE. This matches the fitness change of about 0.04 under fixed question text (Elite ΔF 0.036 fundamental, −0.043 technical).
W2. The improvement is unstable; is it only a restructuring of the initial questions
Revising existing questions is itself part of factor development: we examine whether natural language can serve as the factor definition, so we do not require every descendant question to propose a new mechanism.
W3 & Q3. Future information in model parameters; cutoffs; earlier models or retrieval only
Public information and our own tests put the cutoff of DeepSeek-v4-flash at about May 2025 and that of Qwen3.6-35B-A3B, which writes the summaries, at about March 2025 (now in Appendix C, marked as approximate). Validation (Jul–Sep 2025) and test (Jan–Mar 2026) come after both, so final selection and test scoring should not use memorized information from those periods; development (Apr 2024–Mar 2025) lies inside the models' knowledge, as stated in the appendix. Earlier-cutoff models lack the capability for multi-source retrieval; later-cutoff models compress the available dates, lowering confidence on low-SNR data. The two cannot both be met, so development lies inside the knowledge range while validation and test follow the cutoff, leaving no direct leakage.
The retrieval-only baseline needs a program for mixed inputs, so we report it for reference only. It is strongest on NDX technical (IC 0.070, RankIC 0.071, close to Consolidated NL's 0.059/0.050) and near zero or negative elsewhere: it derives from agent-rewritten questions, and the technical setting is mostly structured market data, where programs have an advantage. Our gains come mainly from fundamental judgments.
W4, Q2 & Q4. Measurement checks and semantic reversal; questions kept; the list
Semantic reversal rewrites a question rather than flipping the sign of a score. The agent sometimes ignores direction, so two phrasings of one idea can give ICs of opposite sign; rather than compare |IC|, we have the agent state the opposite hypothesis when a question shows a persistent negative relation, so a high score means a more positive expected return. Five final questions were reversed: CSI300 technical #1 and #2; NDX fundamental #1 (leverage expansion to deleveraging) and #5 (leverage reversed with a risk constraint); and NDX technical #4 (volume expansion to a 20-day contraction). The rest keep their direction. We keep 12 questions per setting, re-executed on the test-date materials; the full text is in Appendix E.3, Table 11.
On the measurement checks: the factor is the question itself, and it can express any property stated in natural language, not only those a formula can capture. Formula comparison asks whether a score ranks stocks as the matching formula does, so it needs a formula to compare against, and the classical questions have one; most final questions instead combine unstructured sources (filings, news, sell-side text) and involve judgment, so no formula defines their target, which is why we define factors by questions.