acceptodds
Under review as a conference paper at ICLR 2027

From Search to Research: Exploring Search Scaling in Autonomous Quantitative Factor Mining

10 views

Abstract

Inference scaling has been shown to improve large language model (LLM) performance, and this principle naturally extends to autonomous LLM agents through increased search budgets, which we refer to as *search scaling*. Although prior work has characterized the mechanisms, scaling behavior, and performance limits of LLM inference scaling, much less is known about these questions in autonomous research. Therefore, we investigate how search scaling affects research performance and what mechanisms drive these gains using 50 quantitative factor-mining tasks grounded in financial research reports. Each task requires an agent to carry out an end-to-end research loop, from interpreting a hypothesis and implementing it in code to evaluating and iteratively refining the resulting factor. Across nine models, we examine how model capability, search depth, and search organization shape factor quality by tracing performance across varying budgets, transferring intermediate research states between models, and comparing different search strategies. We find that (1) initial performance is more strongly associated with model capability, while deeper search can narrow cross-model gaps; (2) early research state can materially shape final influence; and (3) parallel search outperforms sequential search under the same iteration budget, consistent with benefits from broader coverage of the search space. Further trajectory analysis shows that higher-performing models more effectively diagnose failures, revise search directions, and preserve the intended economic hypothesis when selecting candidates. These findings suggest that future progress in autonomous research will require stronger models together with adaptive policies for deploying test-time computation throughout the research process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agent, search scaling, test-time scaling, AI R&D, alpha mining, quantitative research

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7mYNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies “search scaling”: how giving an LLM research agent more iterations changes the quality of its output. The testbed is 50 factor-mining tasks drawn from Chinese A-share sell-side reports. In each task the agent implements a factor from an economic hypothesis, backtests it on 2013–mid-2024 data, and revises it over 10 iterations inside a shared Claude Code harness. A checkpoint after each iteration is evaluated on held-out data (July 2024–July 2026), and the agent never sees those results. Factors are scored on a composite of net Sharpe, decile monotonicity, and hypothesis alignment, with alignment rated by a GPT-5.6 auditor. Nine models are compared.

The paper reports five main findings:

  1. External capability (AAII) predicts early quality almost perfectly (Spearman 1.00 at b=3) but predicts gains from further search only weakly (0.48).
  2. Grafting a stronger model’s early state onto a weaker continuer beats the reverse order.
  3. Five parallel 2-iteration runs beat one 10-iteration run.
  4. DeepSeek-V4.1-Flash dominates the cost frontier.
  5. In qualitative analysis, stronger agents redirect at the representation level, attribute feedback through ablations, and reject higher-Sharpe candidates that drift from the hypothesis.
Critical strengths
  1. The evaluation setup is clear. Authors included all setups, prompts, and results in the paper.
  2. The cost of runs is clearly presented and analyzed in the paper.
Critical weaknesses
  1. Overall, this is a paper focused on findings. The experiments are mostly practical runs, without raising new claims or inventing new methodologies. Therefore, I do not find this paper to be offering particularly novel research insights.
  2. The central contribution seems to be the new benchmark that is introduced, which is not released by the authors.
  3. The experiments are not completely solid. For example, some of the models evaluated could have been trained on the 2024-2026 market data or reports that are used for evaluation. Contamination risks may arise.
Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

On Data Contamination

Regarding potential data contamination, our evaluation protocol incorporates several safeguards. All source research reports were published by June 2024, whereas the held-out market evaluation period begins in July 2024. Agents only receive development-period feedback, and held-out evaluation results are never returned to them. Furthermore, the backtesting interface enforces point-in-time information availability, and the independent audit checks whether submitted implementations bypass this interface or introduce information unavailable at the signal date. We acknowledge that these safeguards cannot completely rule out prior exposure of evaluated models to publicly available market information during pretraining, particularly when the exact composition of proprietary training corpora is unknown. However, such potential exposure is distinct from direct access to held-out evaluation data during agent execution. We will clarify this residual limitation and the distinction between model-pretraining exposure and evaluation-data leakage.

Reviewer_GWRsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates search scaling for closed‑loop autonomous research agents. It constructs a benchmark of 50 quantitative factor‑mining tasks derived from Chinese‑A‑share financial reports. Agents perform the full research pipeline: hypothesis interpretation, code implementation, backtesting and iterative refinement. The authors run multi‑budget experiments across nine LLMs, together with two controlled interventions: model grafting and sequential‑versus‑parallel search. Key findings: base‑model capability dominates initial performance; deeper search narrows inter‑model performance gaps; early intermediate research states substantially shape final outcomes; parallel search outperforms sequential search under equal iteration budgets. The paper further identifies behavioural differences between high‑ and low‑performing agents from agent trajectories and builds an out‑of‑sample evaluation benchmark with hypothesis‑alignment audit. Limitations include a narrow experimental domain and limited samples for key interventions; generalisability remains open.

Critical strengths
  • Originality: Proposes the clever model‑grafting intervention to disentangle base‑model capability from intermediate research artifacts. Extends test‑time‑scaling research to closed‑loop research agents with real experimental feedback.
  • Experiment quality: Rigorous evaluation protocol with strict out‑of‑sample assessment. Hypothesis‑alignment metric mitigates shortcut solutions. Reproducibility materials are complete.
  • Significance: Highlights that intermediate artifacts and search organisation matter comparably to raw model capability, offering practical insights for agent engineering and the financial‑LLM community.
Critical weaknesses
  • Poor external validity: All experiments are restricted to A‑share factor mining with pre‑given hypotheses (no open hypothesis generation). Key interventions use very few model samples. The scaling fit is trained on limited data without out‑of‑distribution validation.
  • Limitations in behavioural analysis: Relies on manually selected examples without task‑wide quantitative statistics, creating selection‑bias risks which are not explicitly acknowledged.
  • Dataset & presentation gaps: No task‑difficulty stratification analysis; some over‑claimed concepts; unexplained financial jargon and delayed motivation for key interventions harm readability for general ML readers.
Questions for authors

Same as weakness

1 comment
AuthorsOpenReview ↗

On Behavioral Analysis and Task Difficulty

We appreciate the suggestion to provide more systematic quantitative evidence for the behavioral differences. Our current analysis is intended to identify potential mechanisms underlying performance differences, with Appendix E.2 presenting paired research cases illustrating three patterns: search redirection after a plateau, feedback attribution through controlled comparisons, and objective-consistent candidate selection. These cases are grounded in the agents' persistent research memories and corresponding experimental outcomes. We agree that selected examples alone cannot establish the prevalence of these behaviors across the entire task set, and will clarify the qualitative nature of this analysis. Regarding task difficulty, all models and search budgets are evaluated on the same 50 objectives, enabling paired comparisons under a fixed task composition. The benchmark also covers 20 fundamental and 30 technical tasks, spanning multiple signal families and both daily and intraday inputs. Furthermore, Appendix E.1 already reports task-level performance metrics for all 50 tasks across the three search budgets. Although explicit difficulty stratification could provide further insights into heterogeneous search returns, the existing task-level results and shared evaluation objectives provide a basis for examining performance differences without confounding the main comparisons with changes in task composition. We will make these aspects of the experimental design clearer.

Additionally, we will improve our presentation and writing by introducing financial terminology earlier, providing clearer motivation for the controlled interventions, and distinguishing empirical findings from broader implications.

Reviewer_vCjDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates search scaling in autonomous quantitative factor mining using 50 Chinese A-share tasks and nine LLMs under a shared Claude Code scaffold. It evaluates factor quality across continuous research trajectories and conducts model-grafting and sequential–parallel search experiments. The paper reports three findings: 1) deeper search can narrow cross-model capability gaps, 2) early research states influence final performance, and 3) parallel search outperforms sequential search under equal iteration budgets. It also fits an empirical search scaling law

Despite addressing an important question with a careful evaluation protocol, the paper does not establish a sufficiently novel or significant scientific contribution. Its main findings substantially overlap with existing research, while the narrow experimental scope and insufficiently controlled interventions do not support the broader conclusions. These are fundamental limitations of the paper's scientific contribution rather than minor issues of presentation or experimental completeness.

Critical strengths
  1. Understanding how research-time computation scaling interacts with model capability and search strategy is important for designing and evaluating autonomous research agents.
  2. Careful and reusable evaluation protocol. Continuous research trajectories, intermediate checkpoints, independent held-out evaluation, and leakage/alignment audits provide a controlled framework for studying iterative research.
Critical weaknesses
  1. Limited novelty and scientific significance relative to prior work. The main findings substantially overlap with existing research on model-dependent search performance, early research progress, and search-budget allocation (RE-Bench, AIRA-dojo, FML-bench). The paper does not sufficiently distinguish its contributions from these studies or establish what new scientific understanding it provides beyond similar observations in quantitative factor mining.
  2. Overgeneralization from a narrow experimental setting. Although the title explicitly limits the scope to quantitative factor mining, the abstract, discussion, and conclusion present the findings as broader insights into autonomous research. The experiments cover only Chinese A-share factor mining under a single agent scaffold (Claude Code), without validating these findings across other research domains or agent designs. If interpreted strictly within this domain, the broader scientific significance of these observations remains unclear.
  3. Inadequate investigation of the exploration–exploitation trade-off. Finding 3 relies on a single comparison (5×2 parallel vs. 1×10 sequential iterations) across three models. Moreover, prior work shows that the exploration–exploitation trade-off depends on task difficulty, improvement opportunities, model capability, and search policy (AIRA-dojo; FML-bench). The paper neither investigates these dependencies nor isolates the proposed benefit of broader search coverage, making its conclusion insufficiently supported.
  4. Insufficient empirical support for Findings 1. Finding 1 suggests that deeper search narrows cross-model gaps, yet Figure 2 does not show consistent convergence. Table 1 reveals widening gaps for several model pairs (e.g., GLM-5.3 vs. Qwen3.7-Plus), and no aggregate evidence for gap narrowing is provided.
  5. Limited evidence for the proposed scaling law. The scaling law is fitted using only three search depths (3, 6, and 10 iterations), without evaluating whether the proposed relationship holds at larger search budgets.
  • RE-Bench: "RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts." ICML, 2025
  • AIRA-dojo: "AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench." Neurips, 2025
  • FML-bench: "FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics." arxiv 2026
Questions for authors
  1. What new scientific understanding does this paper establish beyond prior work on model-dependent search scaling, early research progress, and search-budget allocation? Which findings constitute genuinely new insights rather than observations of previously studied phenomena in a different domain?
  2. What evidence supports generalizing the findings beyond quantitative factor mining, given that search behavior depends on task structure, model capability, and agent scaffold? If the conclusions are restricted to this specific setting, what broader scientific significance do they provide beyond existing research?
  3. Given prior evidence that the optimal breadth–depth trade-off depends on task characteristics and model capability, how can the paper support its broader conclusion that parallel search outperforms sequential search based on a single comparison in one domain?
1 comment
AuthorsOpenReview ↗

On Exploration Strategy and Cross-Model Scaling

Exploration–exploitation trade-off and parallel search. We appreciate the observation that the optimal balance between exploration and exploitation may depend on task characteristics and model capability. Our experiment is designed to examine a specific and practically relevant allocation choice under a fixed nominal iteration budget: five independent two-iteration searches versus one continuous ten-iteration search. The parallel configuration consistently improves both Score and Sharpe across all three tested models that are selected to represent LLMs of different capability levels: DeepSeek-V4.1-Flash (89.80 vs. 83.44), Doubao Seed 2.1 Pro (73.00 vs. 67.97), and MiniMax-M3 (61.57 vs. 55.99). Thus, the improvement is consistent across all six model–metric comparisons. We acknowledge that this experiment does not establish the universal optimality of parallel search across different task difficulties, model capabilities, or budget allocations. In particular, the observed advantage reflects the combined effect of exploring multiple initial directions and selecting candidates using development feedback. Nevertheless, the consistent results provide empirical evidence that, under our experimental setting, broader exploration can be more beneficial than extending a single research trajectory. We will refine the wording of this finding to reflect its empirical scope while retaining its practical implication for search-budget allocation.

Evidence for Finding 1. We would like to clarify that our finding concerns the possibility of narrowing cross-model performance gaps through deeper search, rather than uniform convergence across all model pairs. For example, GLM-5.3 starts below Claude Opus 5 at three iterations (67.15 vs. 75.45) but slightly surpasses it at ten iterations (92.65 vs. 92.50), providing a concrete instance of gap narrowing. We acknowledge that this pattern does not hold for every model pair, as the reviewer correctly points out. This heterogeneity is also consistent with our broader observation that the returns to additional search vary across models. Furthermore, our capability analysis reveals that AAII has a strong correlation with initial performance (Spearman's ρ = 1.00), but a substantially weaker correlation with the Sharpe improvement from three to ten iterations (ρ = 0.48). This distinction suggests that initial model capability and the ability to benefit from continued research are related but different dimensions. We will clarify that deeper search can narrow certain cross-model gaps instead of implying universal convergence, and provide a more precise interpretation of the heterogeneous performance trajectories.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to Reviewers: Contributions, Scope, and Scaling Evidence

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate the recognition of the importance of studying research-time computation scaling, as well as the positive assessments of our evaluation protocol and experimental design. We would like to further clarify several aspects of our contributions and provide additional evidence in response to the concerns raised.

  1. Novelty and scientific significance. We agree that understanding the relationship between model capability, search budgets, and research performance is an important direction shared with prior work. Our contribution, however, focuses on a distinct setting: closed-loop empirical research, where agents implement hypotheses, execute experiments, interpret feedback, and iteratively revise persistent research artifacts. While RE-Bench, AIRA-dojo, and FML-bench investigate related aspects of AI research agents, our work complements these studies by examining three relatively underexplored questions: (i) how base-model capability relates to both initial research quality and the marginal returns of continued search, (ii) how intermediate research states transfer across models and influence subsequent research, and (iii) how computation should be allocated between independent exploration and sequential refinement. In particular, model grafting provides a controlled intervention into the relationship between research-state quality and model capability. We transfer the accumulated factor, development evidence, and research memory between GLM-5.3 and Qwen3.7-Plus after three iterations, then let the receiving model continue for seven iterations. Starting with GLM-5.3 and continuing with Qwen3.7-Plus achieves a score of 76.41, compared with 72.28 for the reverse schedule, despite using the same model pair and total iteration budget. This result provides evidence that the quality of an early research state has downstream value that cannot be fully explained by the capability of the continuing model. Together with the sequential–parallel intervention, these experiments offer insights into how research states and computation allocation jointly shape research outcomes. We will further strengthen the distinction between our contributions and related work in the revised manuscript.

  2. Experimental scope and generalizability. We appreciate the concern regarding the generalizability of our findings. Our choice of quantitative factor mining is deliberate: it provides executable hypotheses, reproducible experimental feedback, persistent research artifacts, and independently measurable out-of-sample outcomes, making it a suitable testbed for studying closed-loop research. Providing predefined hypotheses is also an intentional design choice that allows us to isolate the research and refinement process from open-ended hypothesis generation, which introduces additional sources of variation. Similarly, using a shared Claude Code harness controls for differences in agent infrastructure, allowing us to study the effects of model capability and search budgets under a consistent execution environment. We acknowledge that experiments across different domains and agent scaffolds would further strengthen external validity. Nevertheless, we believe the current setting provides a meaningful controlled investigation of search scaling, with potential relevance to other research tasks that share similar properties, such as executable feedback, persistent artifacts, and iterative revision. We will make the scope of our empirical claims more explicit and distinguish findings directly supported by our experiments from broader implications for autonomous research.

  3. Empirical scaling law. We would like to clarify that the empirical scaling relationship is fitted using 24 model–budget observations from eight models across three search depths. In addition, Appendix D.4 includes leave-one-model-out validation: each fold fits seven models and predicts all three checkpoints of the held-out model, yielding an aggregate RMSE of 0.193 Sharpe across eight folds. This provides evidence that the fitted relationship captures a cross-model pattern beyond the individual observations used in each training fold. We also report that the fitted relationship explains 92.7% of the variation in the observed model means, with an RMSE of 0.138 Sharpe. We agree that the limited number of search depths restricts our ability to assess extrapolation to substantially larger budgets. Accordingly, we intend the proposed relationship as an empirical characterization of diminishing returns within the observed range, rather than a universal scaling law. We will make this distinction more explicit in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.