- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies “search scaling”: how giving an LLM research agent more iterations changes the quality of its output. The testbed is 50 factor-mining tasks drawn from Chinese A-share sell-side reports. In each task the agent implements a factor from an economic hypothesis, backtests it on 2013–mid-2024 data, and revises it over 10 iterations inside a shared Claude Code harness. A checkpoint after each iteration is evaluated on held-out data (July 2024–July 2026), and the agent never sees those results. Factors are scored on a composite of net Sharpe, decile monotonicity, and hypothesis alignment, with alignment rated by a GPT-5.6 auditor. Nine models are compared.
The paper reports five main findings:
- External capability (AAII) predicts early quality almost perfectly (Spearman 1.00 at b=3) but predicts gains from further search only weakly (0.48).
- Grafting a stronger model’s early state onto a weaker continuer beats the reverse order.
- Five parallel 2-iteration runs beat one 10-iteration run.
- DeepSeek-V4.1-Flash dominates the cost frontier.
- In qualitative analysis, stronger agents redirect at the representation level, attribute feedback through ablations, and reject higher-Sharpe candidates that drift from the hypothesis.
Critical strengths
- The evaluation setup is clear. Authors included all setups, prompts, and results in the paper.
- The cost of runs is clearly presented and analyzed in the paper.
Critical weaknesses
- Overall, this is a paper focused on findings. The experiments are mostly practical runs, without raising new claims or inventing new methodologies. Therefore, I do not find this paper to be offering particularly novel research insights.
- The central contribution seems to be the new benchmark that is introduced, which is not released by the authors.
- The experiments are not completely solid. For example, some of the models evaluated could have been trained on the 2024-2026 market data or reports that are used for evaluation. Contamination risks may arise.
Questions for authors
See weaknesses
On Data Contamination
Regarding potential data contamination, our evaluation protocol incorporates several safeguards. All source research reports were published by June 2024, whereas the held-out market evaluation period begins in July 2024. Agents only receive development-period feedback, and held-out evaluation results are never returned to them. Furthermore, the backtesting interface enforces point-in-time information availability, and the independent audit checks whether submitted implementations bypass this interface or introduce information unavailable at the signal date. We acknowledge that these safeguards cannot completely rule out prior exposure of evaluated models to publicly available market information during pretraining, particularly when the exact composition of proprietary training corpora is unknown. However, such potential exposure is distinct from direct access to held-out evaluation data during agent execution. We will clarify this residual limitation and the distinction between model-pretraining exposure and evaluation-data leakage.