acceptodds
Under review as a conference paper at ICLR 2027

Empirical step reward for RL-based Symbolic Regression

Abstract

TD-based advantage estimation (For example, one step forward return ) can be biased when symbolic regression, because the optimal policy is non-ergodic, and there is only terminal reward( ). For example, dataset choice can be important, and wrong symbol choice can be not recoverable in symbolic regression. On the other hand, Monte-Carlo sampling shows high variance which makes training difficult. Tail distribution can be used(Petersen et al., 2021) to partially mitigate this limitation. In our research, intermediate fitness of the equation is obtained for each step, which enables calculation of empirical reward for each step, defined by improvement of the fitness . Our method is based on virtual stack machine, which maps each symbol to stack state update. Our empirical step reward can be seen as finding useful heuristic segments, which is inspired by traditional genetic programming. Our method was verified using financial factor search task, with adjustment for multiple testing bias. Increased diversity of the symbolic expression set is observed in mean-variance optimized portfolio return.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Credit Assignment, Reward Shaping, Deep Symbolic Regression, Financial Factor Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5DwiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a modeling of the "Empirical Asset pricing" as a deep symbolic regression problem, which allows to use solutions such as Deep Symbolic Regression (DSR) to solve the problem. The authors note that the vanilla version of the DSR framework lack in diversity and thus propose a modification of the framework leveraging reward shaping that would allow to find more diverse solutions and supposedly be more appropriate for the domain.

The paper is very unclear and the current structuring makes the reader wonder about what is the intended contribution for a long time. I had to read pretty much the whole paper undecided on whether if the authors intended to improve deep symbolic regression methods in general providing better diversity in sampling, or if they intended to propose a modeling for the asset pricing domain. It seems to be the latter, but the paper structure isn't appropriate for either of them. I would personally make the introduction mention what are the challenges of the state of the art techniques for this domain, then: (i) why symbolic regression is a good approach to solve a problem, then (ii) what are the gaps for the SOTA in this area so that the algorithms are applicable, then (iii) a detailed description of the problem being solved (regardless if using SR or not), what are the inputs and outputs and how success is measured; (iv) the SR modeling of the problem with detailed description of how to calculate each state feature, then (v) the experimental evaluation and conclusion.

Critical strengths

THe main strength of the paper is presenting a new interesting application for SR. However, the description of the paper is so unclear that the value is not delivered since a third party reading the paper wouldn't be able to replicate the domain.

Critical weaknesses

The main weakness of the paper is that it is extremely unclear to the point the reader struggles to understand what is the intended contribution of the paper. It is not a pleasant read where the structure doesn't help the reader and the paragraphs seem very disconnected (perhaps excessive use of AI for polishing the text?).

Another strong weakness is that the empirical evaluation is not only extremely unclear (where did the financial factor discovery simulation came from?), the evaluation is insufficient to the intended contribution. If the authors intended to show that SR could be used to achieve SOTA in this domain the comparison should be against the SOTA solutions (even if non AI) in this area, it makes no sense to evaluate just the new proposal against vanilla DSR if DSR is not an established solution for this domain. Conversely, if the goal was instead to actually propose an improvement for deep symbolic optimization in general and this domain was just a benchmark chosen (in addition to rewriting the paper so that this contribution makes sense), the paper should have benchmarked the new solution against GA and other SR approaches, as well as evaluating other domains.

Questions for authors
  • It is very intriguing why you chose to use a transformer network when both the original DSR paper, all following works, and the most popular open source implementation of the framework all use RNN as the base network. I would expect at least some ablation to justify using transformers given that this is not the standard for this approach.
3 comments
AuthorsOpenReview ↗

Reply to Official Review of Submission42557 by Reviewer 5Dwi

Dear reviewer,

Thank you for your helpful review.

[ 1. On paper summary ]

We intend to contribute by showing credit assignment can be improved for black box symbolic regression, and the multiple-testing bias can be adjusted.

We focus on black box regression in general, and the finance was used as an example. We do not claim tradable market anomaly.

  • (i) Why SR : Symbolic regression enables continuous learning, which adds additional value over GA.
  • (ii) Gaps for the SOTA : We think that multiple-testing bias need to be adjusted, when we identify gap in financial applications.
  • (iii) Success measure : We are targeting higher R-sq value in blackbox regression of SRBench (Page 4, Table 1), and higher out-sample sharpe of the financial portfolio after multiple-testing bias adjust (Page 5, Table 2), compared to vanilla REINFORCE.
  • (iv) State definition : It is parent and sibling token. (We updated our method to follow DSR)
  • (v) experimental evaluation : 4-seed SRBench (Table 1) and 10-seed financial (Table 2) benchmark. For financial robustness, different dataset and universe was also tested (Table 3 in the appendix) Our conclusion is that credit assignment is meaningful for black-box regression, even after multiple testing bias correction.

We updated our method, and now our SRBench result is based on DSR, and we modified minimal amount of code to test our method. https://anonymous.4open.science/r/token-level-credit-assignment-FBE1/

[ 2. On critical weakness: ]

Thank you for letting me know about the logic and structure problem. I tried my best to completely rewrite the paper, to make it simple and easy to read.

To improve the reproducibility of our financial backtest, we added our backtest result for equations from Alphagen paper (First line of table 2, and table 5.) We refered Qlib for transaction cost implementation for our backtest.

Please note that we updated our claim, thanks to your advice. We do not claim SOTA nor tradable market anomaly or prediction performance in finance.

[ 3. Answer to the question: ]

Currently we updated the model, and we using the same RNN model with well-known DSR paper, because our previous model was too large to claim generalizability.

When reframing the paper for another venue, I suggest you try to transmit to the reader a bit better the intuition behind the changes. If the main purpose is to show that the "credit assignment can be improved", start by saying where and how the credit assignment is a problem in practical SR applications. Likewise, all success metrics should be mapped to practical deficiencies observed in applications that are not solved by the SOTA.

Also, I am not sure the meaning of "Symbolic regression enables continuous learning, which adds additional value over GA." -> it should be shown that the performance of SR is higher than GA to make such claim.

Thank you for your helpful comment that helps me improving in the next time.

I appreciate your concern about the following points.

  1. I need to clarify the point that I want to improve over the SOTA. (I guess, LLM based financial research.)
  2. I need to empirically show that the contribution of RL for diversity loss problem for dynamic optimization problems.

Please feel free to comment, if I misunderstood your point, or there is any additional comments.

Thank you for your help. I'll try to do more meaningful research furter.

Reviewer_uNjrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The background of this paper lies in the problems existing in reinforcement learning (RL)-based symbolic regression: the traditional episodic reward only gives a reward after the expression is fully generated, which leads to severe credit assignment problems and reward sparsity issues. This paper proposes a potential-based reward shaping method. The core innovation of this paper lies in the fact that it no longer waits until the entire expression is generated before giving rewards, but instead gives immediate rewards based on "how much better the current intermediate result is compared to the previous one" at each step of generation. This allows the reinforcement learning agent to better learn which operations are valuable, ultimately generating more diverse and meaningful mathematical expressions. It has been proven effective in financial factor mining tasks.

Critical strengths
  1. Reward sparsity is indeed a common problem in reinforcement learning (RL)-based symbolic regression. The reward shaping method based on potential functions proposed in this paper is highly innovative and can generate more diverse and meaningful mathematical expressions.
  2. Use the reverse Polish notation to represent mathematical expressions, making it easier to obtain intermediate subexpressions
Critical weaknesses

The reward based on potential functions proposed in this paper seems difficult to generalize to other fields. In the field of financial quantification, each subexpression is a complete factor signal, which can be evaluated. However, in other fields, subexpressions may be meaningless.

Questions for authors

The author should elaborate on the universality of the proposed method and it is recommended to include a discussion on limitations. If possible, it is suggested to revise the paper title by adding information that restricts it to the field of financial quantification.

2 comments
AuthorsOpenReview ↗

Reply to Official Review of Submission42557 by Reviewer uNjr

Dear reviewer,

Thank you for your helpful review.

[ 1. On critical weaknesses : ]

To check out the generalizability, we added benchmark for exact equation recovery and black box optimization.

  • For exact equation recovery task, we could not find evidence of meaning in subexpressions, just like you pointed out. (Page 4, Table 1, Nguyen dataset)
  • On the other hand, token level credit assignment (our advantage modulation or 'AM') shows higher R-sq for black-box regression task.

In addition to that, to make it possible to generalize to other fields,

  • Our potential definition was changed from sharpe to conditional variance, which can be more generally applied.

[ 2. Answer to the question ]

Thank you for the suggestion about the research scope.

  • We changed the title to specify our scope to formulaic factor, which is a kind of black-box regression.
  • Our method does not apply to exact equation recovery task (for example mathematical induction), and this limitation was added to the paper.

Thank you for the detailed response. The authors have adequately addressed my concerns regarding the method's universality and limitations. The revision of the title and the inclusion of the discussion on limitations are well-received. I have no further questions.

Reviewer_kxVUOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a step-level reward for reinforcement-learning-based symbolic regression. Using a virtual stack machine, it evaluates intermediate sub-expressions and defines the reward through changes in a potential function based on annualized Sharpe ratio.

Critical strengths

Strengths

  1. The virtual-stack formulation provides a simple way to evaluate intermediate symbolic expressions.
  2. The paper is transparent about several limitations of its financial backtest, including survivorship bias and the omission of transaction costs.
  3. Controlling the false discovery rate is appropriate for the multiple-testing setting.
Critical weaknesses

Weaknesses:

  1. The main comparison is confounded: the episodic-reward baselines use REINFORCE, whereas the proposed method uses PPO. Sampling temperatures also differ. Therefore, the reported improvement cannot be attributed specifically to reward shaping.
  2. The evaluation uses only one financial dataset and reports no standard symbolic-regression benchmarks, multiple random seeds, uncertainty estimates, or reward-component ablations.
  3. The in-sample/out-of-sample split and several implementation details are insufficiently specified.
  4. Figure 5 does not directly establish the claimed reduction in return variance or credit-assignment difficulty.
Questions for authors

Please refer to the weaknesses.

4 comments
AuthorsOpenReview ↗

Reply to Official Review of Submission42557 by Reviewer kxVU

Thank you for your helpful review.

[ On our weaknesses ]

  1. To remove confounding effects, we updated our method to follow DSR with only reward shaping difference (page 3, line 130).
  2. SRBench benchmark (Page 4, table 1, 4-seed) was added, and various financial dataset( Page 5, Table 2 is price data in CSI300 market, and Page 6, Table 3 for fundamental and price data in US top 150 market )
  3. We follow DSR implementation for OS/IS split, and 75%/25% split for other SRBench tests. For financial data (CSI 300), test period is 2019/01/01 to 2021/12/31, which is same setting with alphagen(Yu, 2023)
  4. We updated our claim, and our claim is 'R-sq increase effect by our proposed method, for general black-box symbolic regression'.

Thanks for the updates. The added benchmarks, multiple seeds, and standard deviations address much of my concern about the evaluation. The narrower claim about predictive performance also makes sense.

A few details still need clarification: were sampling temperatures and other training settings kept the same? What β did you use, and how was it chosen? Please also give the full financial training/test periods. Table 3 says FDR 5%, while the text and footnote say 50%—which one was used?

Overall, this is a substantial improvement over the initial version.

Thank you for letting me know important points.

[1]

Temperature is 1 for original DSO(DSR) code, for both training and testing.

Our method also uses temperature 1.

( We added make_policy wrapper dso.policy.policy.py which does not modify the sampling setting )

The code can be check from the following link : https://anonymous.4open.science/r/token-level-credit-assignment-FBE1/

[2]

The financial training and test period is

  • CSI 300 In-sample : 2009-01-01 to 2018-12-31 , CSI 300 Out-sample : 2019-01-01 to 2021-12-31
  • US equity In-sample : 2015-02-02 to 2018-12-31, US equity Out-sample : 2019-01-02 to 2021-11-30

Please note that we didn't add gap between the in-sample and out-sample set, so there is a risk of data leakage.

[3]

  • Table 3(US equity) uses FDR 50%, and 5% is the column is the typo.
  • Table 2(CSI300) uses FDR 5%

The reason of using FDR 50% for table 3 is, data characteristics difference. (shorter period and including fundamental data)

When we tried FDR 5% for table 3, only one or no signal was selected. So, we adjusted it to FDR 50% for Table 3.

We think this result make sense, becase we need more statistical significance when we use only price data.

[ * ]

Thank you for your time and your generous considerations.

When I try next time, I will state experimental details, and I will explain reason of different conditions.

Thanks for clarifying the setup. This addresses most of my concerns, and I’ll raise my score from 1 to 2. Please specify the value of beta, explain the FDR setting and possible data leakage, and include these discussion points in the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.