acceptodds
Desk rejected at ICLR 2027

FactorRSI: LLM-Guided Temporal Factor Mining through Recursive Self-Improvement

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long behavioral histories can contain predictive signals that differ in temporal scope, sample applicability, and modeling requirements. Manually identifying and combining these signals is costly. LLM-guided factor proposals reduce this effort, but a fixed search policy can continue to allocate resources to unproductive directions as evidence accumulates. We formulate LLM-Guided Factor Evolution as joint search over executable sample conditions, temporal windows, and encoder families, followed by fusion of applicable experts. We then introduce RSI (Recursive Self-Improvement)-Guided Factor Evolution, which adapts the policy directing this search. An outer controller uses search outcomes and VALID-monitor feedback to revise search allocation and proposal preferences, retaining or rolling back each update after validation. Across five datasets and three LLM backbones, RSI-on improves 46 of 48 reported endpoint metrics over RSI-off; the five Q3.5-397B comparisons use 8.2–77.8% fewer input tokens. Factor analyses find useful window–encoder combinations, strong descendants especially from crossover, and gains from combining applicable experts.

Primary area:
interpretability and explainable AI
Keywords:
large language model, temporal factor, agent self improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S6bxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FactorRSI, a framework for LLM-guided temporal factor discovery with adaptive search policies. It represents each factor as an executable combination of a sample applicability condition, a temporal window, an encoder family, and a training configuration. Candidate factors are generated and refined through exploration, mutation, and crossover, while multiple applicable factor experts are combined for prediction. The main contribution is an additional RSI-based controller that uses validation feedback to adapt search operator allocation and factor proposal preferences. Policy modifications are evaluated, and either retained or rolled back based on validation performance. Experiments are conducted on five datasets spanning financial forecasting, retail demand prediction, and traffic forecasting, using three LLM backbones. The authors report improvements over fixed-policy factor evolution and several existing factor mining approaches, along with reduced input-token consumption at the reported search endpoints.

Critical strengths
  1. The factor representation is explicit and executable. It combines sample applicability, temporal scope, and encoder choice. This representation permits specialized experts to contribute to the same prediction. It also makes their applicability easy to inspect.
  2. The framework supports several prediction tasks. The experiments cover financial signals, retail demand, and traffic speed. Both RSI settings use the same basic factor and fusion pipeline. This comparison provides a useful starting point for assessing policy adaptation.
Critical weaknesses
  1. TEST feedback and result selection limit the generalization claim. Appendices B, F, and L disclose aggregate TEST feedback to the inner generator on four datasets. The outer controller's validation-only input does not make the complete search independent of TEST. Table 1 reports performance extrema. Appendix J also states that the best metrics need not share one checkpoint. Thus, the reported metric tuples may not describe one model selected without TEST information. A causal online protocol could be valid, but the paper does not sufficiently establish its feedback timing and scoring procedure.
  2. The comparisons do not establish the controller's reliable benefit or efficiency. Several endpoint comparisons use different search horizons. For example, Table 5 compares CSI500 at rounds 80 and 250. Appendix J combines separate run batches without a verified common horizon or independent-seed variation. Policy acceptance also compares changing factor populations. The system retains generated factors after it reverses a policy change. These results do not isolate the benefit of policy adaptation. They also do not establish an advantage under equal budgets. Input-token savings do not necessarily imply lower total cost. On PEMS-BAY, RSI-on has slightly higher elapsed time.
Questions for authors
  1. For the four datasets that return TEST feedback, what information becomes available before each prediction? Does each result affect only later predictions? Can you provide records that verify this order?
  2. Which rule selects the reported checkpoint for each method? Do all metrics in each table cell come from that checkpoint? If not, can you report one checkpoint selected using validation alone?
  3. Do you have repeated runs with equal resource budgets and uncertainty estimates? What stopping rules produced the different endpoint rounds?
  4. Do you have comparisons that start from the same factor pool and test competing policy changes with equal additional budgets? Such evidence could change the assessment of the controller.
  5. Appendix F defines M5 IC as (1/(1+WRMSSE))(1/(1+\mathrm{WRMSSE})). Several entries in Tables 1 and 4 do not satisfy this relationship beyond rounding. For example, Table 4 reports StockMixer WRMSSE of 2.036 and IC of 0.341. The formula gives approximately 0.329. Do the paired values use different evaluation configurations or checkpoint sets? Please explain these differences and correct the metric definition or reported values, as applicable.
3 comments

We organize our response around two aspects that address the reviewer's concerns collectively.

1. Validation-Only Protocol and Checkpoint Selection (Weakness 1, Questions 1–2)

Under the revised protocol, all decision-making components—model fitting, expert selection, ensemble construction, checkpoint selection, and RSI policy updates—use only TRAIN and VALID information. TEST metrics are computed only after the checkpoint is frozen and are never returned to the search process.

The information flow is strictly causal: at each round, the model and controller access only the current round's VALID results and all prior history. The TRAIN/VALID/TEST splits are partitioned in strict chronological order with no overlap, and dataset-specific target-realization purges are applied at every boundary, precluding look-ahead bias by construction. The exact split boundaries are documented in Table 2 of the Appendix.

We apply a single, explicit checkpoint-selection rule uniformly across all methods and datasets: the checkpoint with the highest VALID ICIR is selected, and TEST IC, ICIR, and RankIC are all read from that same factor-pool snapshot. The three reported values correspond to one checkpoint rather than separate metric-wise maxima, eliminating any possibility of cherry-picking.

2. Controller Benefit, Efficiency, and Reproducibility (Weakness 2, Questions 3–4)

Equal-budget comparison. We replaced the original unequal-endpoint comparison with a common-budget analysis, where the budget is defined as cumulative inner-loop token consumption (input + output). At 25%, 50%, 75%, and 100% of the shared token budget, RSI-on achieves higher TEST ICIR at 15 of 16 fixed-budget checkpoints and yields a higher TEST-ICIR–token AUC on all four datasets (see our response to Reviewer wcPH, M3-a). A separate comparison against Random, GP-BO, and UCB1 on CSI300 under an equal candidate-evaluation budget of 254 fits across three seeds confirmed that RSI-on achieved the best mean TEST IC, ICIR, and RankIC while completing the search in roughly half the wall-clock time (see our response to Reviewer 4nii, Weakness 2).

Isolation of policy benefit. The final implementation adopts transactional rollback: when a policy update is rejected, the system reverts both the policy parameters and all experts generated under the rejected policy, cleanly isolating the policy's contribution from incidental pool growth. We further ran a validation-only counterfactual experiment on CSI300 at three policy-update points (R40, R60, R100) across three seeds, where the proposed and incumbent policies each continued from the identical expert pool for 20 candidate evaluations. The proposed policy achieved higher TEST ICIR in 7 of 9 matched pairs, confirming that policy replacement provides a consistent benefit beyond pool growth (see our response to Reviewer 4nii, Weakness 4). At comparable TEST performance, RSI-on also uses 19.1–53.8% fewer contributing expert slices, producing more compact ensembles that reduce inference cost (see our response to Reviewer wcPH, M3-a).

Stopping rule and repeated runs. All methods share the same stopping criterion: the search continues until performance plateaus, with a hard cap of 500 rounds. RSI-on and RSI-off reach their best VALID ICIR at different rounds because the self-improvement mechanism alters the search trajectory; the differing endpoint rounds reflect this natural variation rather than asymmetric budget allocation. The main-table results are now reported as means over three independent seeds using Qwen3.5-397B, with each seed's checkpoint selected by the highest VALID ICIR. Multi-seed results for all base model variants will be included in the revised manuscript.

Thanks for your clarification. My concerns are addressed. I have updated my score from 2 to 3.

Thank you for recognizing the strengths of our approach and for providing constructive suggestions. Your suggestions will help us further strengthen and complete our work. We will refine the revised manuscript before the deadline.

Reviewer_4niiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FactorRSI, a method that uses an LLM to search for predictive experts. Each expert is defined by a sample-selection condition, a historical window, and a model type. The method combines these experts to make predictions and uses an outer LLM controller to adjust the search strategy based on validation results. The authors evaluate the method on five datasets with three LLM backbones and report gains over a fixed search strategy.

Critical strengths
  1. The method searches over sample groups, historical windows, and model types together, which is a reasonable way to capture different patterns in the data.
  2. The paper clearly separates the process of finding better experts from the process of adjusting the search strategy.
  3. The authors evaluate the method across several domains and LLM backbones, providing some initial evidence that the approach is useful.
Critical weaknesses
  1. The novelty is not fully clear to me. The method mainly uses an LLM to make AutoML search decisions and then combines the resulting experts. The paper could better explain how the outer controller differs from existing methods that adapt their search strategies based on feedback.
  2. Following the first point,the paper lacks comparisons with AutoML methods under the same conditions. E.g., Random/bayesian optimization/bandit-based search should use the same search space, feedback, training and combination rules, and resource budget. The comparisons with RSI-off and a random controller do not fully show whether an LLM makes better search decisions.
  3. The current analysis gives some useful examples, but I would like to better understand why the method works. For instance, does removing the experts found later in the search noticeably hurt performance? It would also help to plot how performance changes as the search progresses, using the same budget for RSI-on and RSI-off, and show when the controller changes its strategy. This would make it easier to see which experts and search decisions lead to the gains.
  4. The policy acceptance rule seems reasonable in practice, but an additional comparison would help show whether the new strategy is better. For example, if validation performance rises from 0.30 to 0.32 after a policy change and another 20 search rounds, the gain may partly come from adding more experts. Keeping the old policy and continuing the search might also improve performance. Running both policies from the same expert pool with the same additional budget would make the benefit of the policy change clearer.
Questions for authors
  1. Could you clarify the main contribution beyond LLM-based AutoML search and expert combination? A comparison with an existing AutoML method under the same search space, feedback, and budget would be particularly helpful.
  2. Could you show which discovered experts actually improve the ensemble, and how performance changes over the course of the search? It would be helpful to see these curves alongside the controller’s strategy changes.
  3. Would it be possible to compare the updated and previous search policies from the same expert pool with equal additional budgets? This could help clarify how much improvement comes from the policy update itself.
3 comments

We organize our response around three aspects that address the reviewer's concerns collectively.

1. Architectural Distinction from Existing Adaptive Search Methods (Weakness 1)

FactorRSI differs from existing adaptive search methods in both scope and mechanism. Standard approaches such as bandit-based operator selection or Bayesian optimization treat search hyperparameters as independent arms or continuous variables and adapt them based on scalar reward signals. In contrast, FactorRSI's outer controller jointly adapts the operation mixture, mutation distribution, parent diversity, encoder allocation, and temporal/semantic search directions—treating these as a coherent, co-evolving policy rather than isolated decisions. Furthermore, unlike ReEvo and LLaMEA, which evolve standalone heuristic or algorithm programs, FactorRSI co-evolves a persistent temporal-expert ensemble alongside the policy that generates subsequent experts, evaluating each candidate policy through a multi-step VALID-only rollout. This tight coupling between policy adaptation and ensemble construction is the key architectural distinction from prior work.

2. Empirical Validation of the Controller (Weaknesses 2 & 4)

Controlled comparison with AutoML baselines. We conducted a controlled comparison on CSI300 under a validation-only protocol in which only the search strategy varies; all other factors—search space, data splits, VALID feedback schedule, training and fusion rules, and candidate-evaluation budget—are held constant. Each method (full RSI-on, random search, GP-BO, and UCB1) started from an empty slice pool and ran until 254 candidates had been successfully fitted across three independent seeds.

MethodTEST ICTEST ICIRTEST RankICMean Wall Time (h)
Full RSI-on0.038997 ± 0.0026110.260954 ± 0.0163690.030199 ± 0.0025008.23
Random0.036583 ± 0.0002830.237765 ± 0.0036140.026216 ± 0.00005715.84
GP-BO0.036415 ± 0.0016400.244746 ± 0.0049030.026670 ± 0.00033918.34
UCB10.037385 ± 0.0007030.245129 ± 0.0052230.026672 ± 0.00090315.13

RSI-on achieved the best mean TEST IC, ICIR, and RankIC, outperforming every baseline on all three metrics in two of three runs and on RankIC in all three, while completing the search in roughly half the wall-clock time.

Counterfactual isolation of policy benefit. To disentangle the controller's contribution from incidental pool growth, we ran a validation-only counterfactual experiment on CSI300 at three policy-update points (R40, R60, R100) across three seeds. At each point, we froze the current expert pool and search state, then ran both the proposed policy and the incumbent policy forward for 20 additional candidate evaluations from the identical starting point. Across nine matched pairs, the proposed policy improved mean TEST IC by 0.025%, ICIR by 0.00426, and RankIC by 0.00671 relative to the incumbent, and achieved higher TEST ICIR in 7 of 9 pairs. These results confirm that policy replacement provides a consistent positive benefit beyond what would be obtained from simply continuing with the old policy.

3. Understanding the Search Dynamics (Weakness 3)

Expert removal analysis. On CSI300, we examined intervals with observed TEST ICIR increases and conducted systematic ablations by progressively removing 25%, 50%, 75%, and 100% of the newly added contributing experts. As a control, we also removed the same number of randomly sampled older experts (averaged over 20 draws).

Search IntervalOriginal ICIR GainNew ExpertsRm 25%Rm 50%Rm 75%Rm 100%Rm Same Count (Old)
R20→R300.0070232−0.00161−0.00281−0.00432−0.00702−0.00102
R40→R500.0045942−0.00049−0.00232−0.00364−0.00489+0.00100
R60→R700.0022433−0.00072−0.00125−0.00148−0.00224−0.00037
R260→R2700.0018327−0.00043−0.00090−0.00151−0.00183+0.00030

In all intervals, progressively removing new experts produced monotonically increasing ICIR losses; removing 100% erased essentially the entire gain. In four of five intervals, the loss from removing new experts exceeded that from removing the same count of older experts. These results confirm that experts discovered later in the search contribute directly and disproportionately to performance gains.

Search progression curves. Round-by-round TEST metric trajectories for RSI-on and RSI-off on all datasets are plotted in Figure 7 of Appendix L under the same token budget, showing when the controller adapts its strategy and how each policy change translates into improvement.

Thanks for the response. I think it partially addresses my concerns. First, the expert removal experiment and the comparison of the two policies from the same starting point address points 3 and 4, but I still have two questions.

  1. I understand how FactorRSI changes its search strategy. Existing AutoML methods also adjust what they try next based on earlier results. Furthermore, why were classic methods such as GP-BO and UCB1 chosen as baselines instead of SOTA methods?

  2. FactorRSI did slightly better on CSI300 when each method tried a set of candidates. However, my original question also asked about “similar budgets,” so how much did each run cost, including LLM calls and model training? I was wondering whether FactorRSI would still come out ahead if both methods had similar budgets, or what the trade-off would be.

Thanks for your feedback. We organize our response around three aspects that address your concerns collectively.

Interpretable Temporal Factor Mining

FactorRSI is designed as an interpretable temporal factor-mining framework, rather than only an LLM-based AutoML controller. Each discovered slice explicitly specifies the behavioral condition, historical window, prediction model, and lineage. For example, a slice may select samples with negative 20-day returns and above-average trading volume, and use a 60-day GRU to predict the next period. Such a representation makes the signal’s temporal behavior and decision path auditable, which is particularly important in financial applications.

Distinction from Existing Search Methods

Table 1 compares FactorRSI with FactorMiner, QuantaAlpha, and AlphaAgent, which target the same interpretable factor-mining setting. FactorRSI differs in two respects. First, it jointly uses heterogeneous temporal predictors instead of relying mainly on conditional sample filtering followed by a single strong tree model, allowing complementary models to capture different temporal dependencies. Second, it introduces an outer policy loop that adapts how the inner search is conducted. Based on the VALID trajectory, the outer loop can modify the operation mixture, mutation and crossover distributions, parent diversity, model allocation, temporal search direction, and prompt template. Thus, the outer loop optimizes the search policy itself, rather than only choosing the next candidate within a fixed search procedure.

Budget-Matched Trajectory Comparison

We thank the reviewer for raising the issue of budget fairness. We agree that comparing only the final performance after a fixed number of candidate evaluations does not fully reveal the performance–cost trade-off. We therefore performed a trajectory-level comparison. For each metric, we first identified the highest TEST value achieved by RSI-off over its complete trajectory, and then located the first RSI-on checkpoint that strictly exceeded this value. For example, on CSI300, RSI-off reached its best IC of 0.043373 only at R500, whereas RSI-on exceeded it at R150 with an IC of 0.043511. The corresponding RankIC cost comparison is shown below:

DatasetRSI-off best RankICRSI-on first exceeding checkpointLLM calls (on/off)Total tokens (on/off)
CSI3000.037029 at R2300.037279 at R190195 / 2331.765M / 2.145M
CSI5000.027672 at R1400.027760 at R7073 / 1410.674M / 1.295M
Numerai0.047498 at R1200.047611 at R120124 / 1200.741M / 0.686M
PEMS-BAY0.739023 at R1000.739163 at R4041 / 1000.197M / 0.472M

The reference budget is defined at the best observed RSI-off checkpoint. As evolution proceeds, RSI-off continues to accumulate slices, so its pool and effective ensemble cost increase, while RSI-on dynamically adjusts its effective ensemble budget according to validation performance and may either increase or decrease it. In the observed trajectories, RSI-on has already surpassed the RSI-off reference performance at a substantially lower cost. After this point, RSI-off continues to incur increasing cost while approaching a performance plateau, whereas RSI-on continues to improve without a pronounced decline while regulating its effective ensemble size. Given the limited time available during the rebuttal period, we will make this behavior explicit in the revised version by plotting both the number of slices participating in the ensemble and the total slice-pool size over evolution rounds, which will allow you to directly assess the trade-off between predictive performance and effective budget.

These results show that RSI-on reaches a strictly higher value than the best observed RSI-off performance earlier and with fewer LLM interactions and tokens on CSI300, CSI500, and PEMS-BAY. Numerai exhibits near-parity. Overall, the trajectory-based analysis shows that FactorRSI generally achieves higher performance with a smaller effective search and ensemble budget, while continuing to improve after surpassing the RSI-off reference level. Detailed results are provided in Appendix L, Figure 7.

Reviewer_wcPHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes FactorRSI, an LLM-guided framework for discovering temporal factors. Each factor is an executable sample condition, a temporal window, and an encoder, and is evolved through exploration, mutation, and crossover. Applicable experts are fused by validation-based reliability weights and blended with a global predictor. An outer RSI controller uses VALID feedback to adjust search allocation and proposal preferences, keeping or rolling back each update. On five datasets with three LLM backbones, RSI-on improves 46 of 48 metrics over RSI-off with fewer input tokens.

Critical strengths
  1. A clear factor representation with direct empirical support. Each factor jointly specifies applicable samples, history window, and encoder, and conditional experts are fused per sample. Figure 4 and Appendix M show that the best encoder differs across (k, L) conditions and that performance is not monotonic in window length. This directly motivates joint search and is the paper’s main distinction from formula-only LLM factor mining.

  2. A transparent method interface. The appendix specifies the condition grammar, the full action allowlist, the prompt structure, and the rollback rule, and the ablations include a random-controller variant. This makes the method easy to inspect and build on.

Critical weaknesses

Major

M1. On four of five datasets, TEST feedback enters the search loop. Appendices B, F, and L state that, on CSI300, CSI500, Numerai, and PEMS-BAY, the inner-loop LLM receives aggregate TEST metrics. Results on these datasets therefore cannot be read as evaluation on an independent final holdout. If the protocol is meant as online evaluation, the manuscript does not sufficiently document how predictions are frozen, when feedback is released, or how cumulative performance is computed. M5, where TEST results are not returned to the LLM, still depends on how its reported results were selected.

M2. The result-selection rule is unclear, and no uncertainty is reported. Table 1 reports results “at performance extrema” without specifying the selection rule. For the ablations, Appendix J explicitly states that no common horizon or shared checkpoint was established. No independent-seed repetitions are reported, while many gaps are small, for example CSI300 IC 0.044 vs. 0.045 and CSI500 IC 0.030 vs. 0.031. The “46/48” count is a valid descriptive summary but not independent statistical evidence: IC, ICIR, and RankIC share the same predictions, and M5’s IC is a deterministic transform of WRMSSE.

M3. The efficiency advantage is not fairly established. Endpoint rounds differ substantially between arms: 80 vs. 250 on CSI500, 50 vs. 250 on Numerai, and 50 vs. 240 on M5 (Table 5). Different rounds would support efficiency if RSI reached a pre-specified target earlier. However, the paper does not disclose the stopping limits, how endpoints were determined, or whether runs completed as planned. Metric-per-input-token also omits output tokens and expert training cost. On PEMS-BAY, input tokens fell but total elapsed time did not (43.7h vs. 43.2h).

M3. RSI’s contribution is not isolated, and its distinction from prior work is unclear. Attribution: Rollback restores the policy but keeps newly generated experts, so the acceptance rule cannot separate a better policy from pool growth or noise. Positioning: The self-improvement is limited to bounded search hyperparameters and prompt preferences, which is close to adaptive operator selection and reflective LLM search. The paper does not clarify its technical distinction from, or show an advantage over, bandit-based operator selection, ReEvo, or LLaMEA.

Minor

m1. Several broken references (“Section ??”);

m2. Typos in Figures 1 and 2 (“Widows”, “Crssover”, “predication”);

m3. Figures 14 to 16 are unreadable at their current size;

Questions for authors
  1. On CSI300, CSI500, Numerai, and PEMS-BAY, exactly which TEST information reaches the inner loop, when is it released, and are predictions frozen beforehand? Can the main results be re-reported on a test set fully inaccessible during search?

  2. What rule selects the “performance extrema” in Table 1, and do all metrics in a row come from the same checkpoint? Under a pre-specified VALID selection rule with multiple seeds, how large is the RSI-on/off gap?

  3. What stopping limits applied to each run, how were endpoint rounds determined, and did all runs complete as planned? What are the results under equal rounds or equal total compute, including expert training?

  4. Does the CSI300 and CSI500 setup match the official Qlib protocol in data version, label definition, normalization, and training configuration? If so, why are LightGBM, XGBoost, and DoubleEnsemble far below the official results?

1 comment

We organize our response around three aspects that address the reviewer's concerns collectively.

1. Validation-Only Protocol and Checkpoint Selection (M1, M2)

We acknowledge that the original protocol—in which aggregate TEST metrics were fed back to the inner loop—cannot support an independent-holdout claim. We have therefore replaced all such results with a validation-only protocol: both the inner factor generator and the outer RSI controller now receive only TRAIN and VALID signals throughout the entire search process. TEST data are never observed by any decision-making component.

We apply a single, explicit checkpoint-selection rule uniformly across all methods and datasets: the checkpoint with the highest VALID ICIR is selected, and TEST IC, ICIR, and RankIC are all read from that same factor-pool snapshot. The three reported TEST metrics therefore correspond to one checkpoint rather than separate metric-wise maxima, eliminating any ambiguity about cherry-picking. The updated results under this protocol are shown below:

DatasetRSI-on CkptRSI-on TEST IC / ICIR / RankICRSI-off CkptRSI-off TEST IC / ICIR / RankIC
CSI3002500.046214 / 0.314399 / 0.0374022300.042714 / 0.280325 / 0.037029
CSI5001600.032822 / 0.285840 / 0.0319721200.030653 / 0.270169 / 0.027627
Numerai1300.060701 / 3.205243 / 0.047718400.057164 / 2.941978 / 0.045981
PEMS-BAY1000.719375 / 6.060509 / 0.7400821100.718651 / 6.002755 / 0.738982

RSI-on outperforms RSI-off across all three TEST metrics on all four datasets. Complete round-by-round trajectories are provided in Figure 7 of Appendix L.

2. Efficiency Under Equal Budgets and Ensemble Compactness (M3-a)

We replaced the original unequal-endpoint comparison with a common-budget analysis. The budget is defined as cumulative inner-loop token consumption (input + output). At 25%, 50%, 75%, and 100% of the shared token budget, each arm is assigned the highest VALID ICIR checkpoint observed up to that point, and the corresponding TEST ICIR is reported.

DatasetBudget at 25/50/75/100% (M)25% (on/off)50% (on/off)75% (on/off)100% (on/off)AUC (on/off)
CSI3001.31/2.43/3.55/4.68.2987/.2715.3144/.2803.3144/.2803.3144/.2803.3048/.2763
CSI5000.49/0.79/1.08/1.38.2504/.2463.2639/.2633.2657/.2667.2738/.2702.2585/.2564
Numerai0.26/0.40/0.54/0.692.968/2.9423.015/2.9423.062/2.9423.150/2.9423.022/2.936
PEMS-BAY0.19/0.28/0.38/0.475.984/5.9256.020/5.9776.051/5.9776.052/6.0026.013/5.958

RSI-on achieves higher TEST ICIR at 15 of 16 fixed-budget checkpoints and yields a higher TEST-ICIR–token AUC on all four datasets.

At comparable TEST performance, RSI-on also produces more compact ensembles. Comparing checkpoints with closely matched TEST metrics, RSI-on uses essentially the same number of contributing expert slices on CSI300, but 19.1%, 27.6%, and 53.8% fewer on CSI500, Numerai, and PEMS-BAY, respectively—reducing inference cost without sacrificing predictive quality.

3. Controller Attribution and Distinction from Prior Work (M3-b)

Transactional rollback. The final implementation adopts transactional rollback: when a policy update is rejected, the system reverts both the policy parameters and all experts generated under the rejected policy.

Architectural distinction. FactorRSI differs from fixed-arm operator-selection methods in that it jointly adapts the operation mixture, mutation distribution, parent diversity, encoder allocation, and temporal/semantic search directions as a coherent, co-evolving policy rather than independent arms. Unlike ReEvo and LLaMEA, which evolve standalone heuristic or algorithm programs, FactorRSI co-evolves a persistent temporal-expert ensemble alongside the policy that generates subsequent experts, evaluating each candidate policy through a multi-step VALID-only rollout.

Controlled comparison with AutoML baselines. To quantify this distinction empirically, we compared RSI-on against Random, GP-BO, and UCB1 on CSI300 under an equal budget of 254 completed candidate evaluations across three independent seeds, with all other factors held constant:

MethodTEST ICTEST ICIRTEST RankIC
Full RSI-on0.038997 ± 0.0026110.260954 ± 0.0163690.030199 ± 0.002500
Random0.036583 ± 0.0002830.237765 ± 0.0036140.026216 ± 0.000057
GP-BO0.036415 ± 0.0016400.244746 ± 0.0049030.026670 ± 0.000339
UCB10.037385 ± 0.0007030.245129 ± 0.0052230.026672 ± 0.000903

RSI-on outperforms UCB1 by 4.31% in TEST IC, 6.46% in ICIR, and 13.22% in RankIC, and completed the search in roughly half the wall-clock time of all baselines. These results confirm that RSI's joint policy adaptation provides a meaningful advantage over standard AutoML alternatives.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.