- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FactorRSI, a framework for LLM-guided temporal factor discovery with adaptive search policies. It represents each factor as an executable combination of a sample applicability condition, a temporal window, an encoder family, and a training configuration. Candidate factors are generated and refined through exploration, mutation, and crossover, while multiple applicable factor experts are combined for prediction. The main contribution is an additional RSI-based controller that uses validation feedback to adapt search operator allocation and factor proposal preferences. Policy modifications are evaluated, and either retained or rolled back based on validation performance. Experiments are conducted on five datasets spanning financial forecasting, retail demand prediction, and traffic forecasting, using three LLM backbones. The authors report improvements over fixed-policy factor evolution and several existing factor mining approaches, along with reduced input-token consumption at the reported search endpoints.
Critical strengths
- The factor representation is explicit and executable. It combines sample applicability, temporal scope, and encoder choice. This representation permits specialized experts to contribute to the same prediction. It also makes their applicability easy to inspect.
- The framework supports several prediction tasks. The experiments cover financial signals, retail demand, and traffic speed. Both RSI settings use the same basic factor and fusion pipeline. This comparison provides a useful starting point for assessing policy adaptation.
Critical weaknesses
- TEST feedback and result selection limit the generalization claim. Appendices B, F, and L disclose aggregate TEST feedback to the inner generator on four datasets. The outer controller's validation-only input does not make the complete search independent of TEST. Table 1 reports performance extrema. Appendix J also states that the best metrics need not share one checkpoint. Thus, the reported metric tuples may not describe one model selected without TEST information. A causal online protocol could be valid, but the paper does not sufficiently establish its feedback timing and scoring procedure.
- The comparisons do not establish the controller's reliable benefit or efficiency. Several endpoint comparisons use different search horizons. For example, Table 5 compares CSI500 at rounds 80 and 250. Appendix J combines separate run batches without a verified common horizon or independent-seed variation. Policy acceptance also compares changing factor populations. The system retains generated factors after it reverses a policy change. These results do not isolate the benefit of policy adaptation. They also do not establish an advantage under equal budgets. Input-token savings do not necessarily imply lower total cost. On PEMS-BAY, RSI-on has slightly higher elapsed time.
Questions for authors
- For the four datasets that return TEST feedback, what information becomes available before each prediction? Does each result affect only later predictions? Can you provide records that verify this order?
- Which rule selects the reported checkpoint for each method? Do all metrics in each table cell come from that checkpoint? If not, can you report one checkpoint selected using validation alone?
- Do you have repeated runs with equal resource budgets and uncertainty estimates? What stopping rules produced the different endpoint rounds?
- Do you have comparisons that start from the same factor pool and test competing policy changes with equal additional budgets? Such evidence could change the assessment of the controller.
- Appendix F defines M5 IC as . Several entries in Tables 1 and 4 do not satisfy this relationship beyond rounding. For example, Table 4 reports StockMixer WRMSSE of 2.036 and IC of 0.341. The formula gives approximately 0.329. Do the paired values use different evaluation configurations or checkpoint sets? Please explain these differences and correct the metric definition or reported values, as applicable.
3 comments
Thanks for your clarification. My concerns are addressed. I have updated my score from 2 to 3.
Thank you for recognizing the strengths of our approach and for providing constructive suggestions. Your suggestions will help us further strengthen and complete our work. We will refine the revised manuscript before the deadline.
We organize our response around two aspects that address the reviewer's concerns collectively.
1. Validation-Only Protocol and Checkpoint Selection (Weakness 1, Questions 1–2)
Under the revised protocol, all decision-making components—model fitting, expert selection, ensemble construction, checkpoint selection, and RSI policy updates—use only TRAIN and VALID information. TEST metrics are computed only after the checkpoint is frozen and are never returned to the search process.
The information flow is strictly causal: at each round, the model and controller access only the current round's VALID results and all prior history. The TRAIN/VALID/TEST splits are partitioned in strict chronological order with no overlap, and dataset-specific target-realization purges are applied at every boundary, precluding look-ahead bias by construction. The exact split boundaries are documented in Table 2 of the Appendix.
We apply a single, explicit checkpoint-selection rule uniformly across all methods and datasets: the checkpoint with the highest VALID ICIR is selected, and TEST IC, ICIR, and RankIC are all read from that same factor-pool snapshot. The three reported values correspond to one checkpoint rather than separate metric-wise maxima, eliminating any possibility of cherry-picking.
2. Controller Benefit, Efficiency, and Reproducibility (Weakness 2, Questions 3–4)
Equal-budget comparison. We replaced the original unequal-endpoint comparison with a common-budget analysis, where the budget is defined as cumulative inner-loop token consumption (input + output). At 25%, 50%, 75%, and 100% of the shared token budget, RSI-on achieves higher TEST ICIR at 15 of 16 fixed-budget checkpoints and yields a higher TEST-ICIR–token AUC on all four datasets (see our response to Reviewer wcPH, M3-a). A separate comparison against Random, GP-BO, and UCB1 on CSI300 under an equal candidate-evaluation budget of 254 fits across three seeds confirmed that RSI-on achieved the best mean TEST IC, ICIR, and RankIC while completing the search in roughly half the wall-clock time (see our response to Reviewer 4nii, Weakness 2).
Isolation of policy benefit. The final implementation adopts transactional rollback: when a policy update is rejected, the system reverts both the policy parameters and all experts generated under the rejected policy, cleanly isolating the policy's contribution from incidental pool growth. We further ran a validation-only counterfactual experiment on CSI300 at three policy-update points (R40, R60, R100) across three seeds, where the proposed and incumbent policies each continued from the identical expert pool for 20 candidate evaluations. The proposed policy achieved higher TEST ICIR in 7 of 9 matched pairs, confirming that policy replacement provides a consistent benefit beyond pool growth (see our response to Reviewer 4nii, Weakness 4). At comparable TEST performance, RSI-on also uses 19.1–53.8% fewer contributing expert slices, producing more compact ensembles that reduce inference cost (see our response to Reviewer wcPH, M3-a).
Stopping rule and repeated runs. All methods share the same stopping criterion: the search continues until performance plateaus, with a hard cap of 500 rounds. RSI-on and RSI-off reach their best VALID ICIR at different rounds because the self-improvement mechanism alters the search trajectory; the differing endpoint rounds reflect this natural variation rather than asymmetric budget allocation. The main-table results are now reported as means over three independent seeds using Qwen3.5-397B, with each seed's checkpoint selected by the highest VALID ICIR. Multi-seed results for all base model variants will be included in the revised manuscript.