- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes a modeling of the "Empirical Asset pricing" as a deep symbolic regression problem, which allows to use solutions such as Deep Symbolic Regression (DSR) to solve the problem. The authors note that the vanilla version of the DSR framework lack in diversity and thus propose a modification of the framework leveraging reward shaping that would allow to find more diverse solutions and supposedly be more appropriate for the domain.
The paper is very unclear and the current structuring makes the reader wonder about what is the intended contribution for a long time. I had to read pretty much the whole paper undecided on whether if the authors intended to improve deep symbolic regression methods in general providing better diversity in sampling, or if they intended to propose a modeling for the asset pricing domain. It seems to be the latter, but the paper structure isn't appropriate for either of them. I would personally make the introduction mention what are the challenges of the state of the art techniques for this domain, then: (i) why symbolic regression is a good approach to solve a problem, then (ii) what are the gaps for the SOTA in this area so that the algorithms are applicable, then (iii) a detailed description of the problem being solved (regardless if using SR or not), what are the inputs and outputs and how success is measured; (iv) the SR modeling of the problem with detailed description of how to calculate each state feature, then (v) the experimental evaluation and conclusion.
Critical strengths
THe main strength of the paper is presenting a new interesting application for SR. However, the description of the paper is so unclear that the value is not delivered since a third party reading the paper wouldn't be able to replicate the domain.
Critical weaknesses
The main weakness of the paper is that it is extremely unclear to the point the reader struggles to understand what is the intended contribution of the paper. It is not a pleasant read where the structure doesn't help the reader and the paragraphs seem very disconnected (perhaps excessive use of AI for polishing the text?).
Another strong weakness is that the empirical evaluation is not only extremely unclear (where did the financial factor discovery simulation came from?), the evaluation is insufficient to the intended contribution. If the authors intended to show that SR could be used to achieve SOTA in this domain the comparison should be against the SOTA solutions (even if non AI) in this area, it makes no sense to evaluate just the new proposal against vanilla DSR if DSR is not an established solution for this domain. Conversely, if the goal was instead to actually propose an improvement for deep symbolic optimization in general and this domain was just a benchmark chosen (in addition to rewriting the paper so that this contribution makes sense), the paper should have benchmarked the new solution against GA and other SR approaches, as well as evaluating other domains.
Questions for authors
- It is very intriguing why you chose to use a transformer network when both the original DSR paper, all following works, and the most popular open source implementation of the framework all use RNN as the base network. I would expect at least some ablation to justify using transformers given that this is not the standard for this approach.
3 comments
When reframing the paper for another venue, I suggest you try to transmit to the reader a bit better the intuition behind the changes. If the main purpose is to show that the "credit assignment can be improved", start by saying where and how the credit assignment is a problem in practical SR applications. Likewise, all success metrics should be mapped to practical deficiencies observed in applications that are not solved by the SOTA.
Also, I am not sure the meaning of "Symbolic regression enables continuous learning, which adds additional value over GA." -> it should be shown that the performance of SR is higher than GA to make such claim.
Thank you for your helpful comment that helps me improving in the next time.
I appreciate your concern about the following points.
- I need to clarify the point that I want to improve over the SOTA. (I guess, LLM based financial research.)
- I need to empirically show that the contribution of RL for diversity loss problem for dynamic optimization problems.
Please feel free to comment, if I misunderstood your point, or there is any additional comments.
Thank you for your help. I'll try to do more meaningful research furter.
Reply to Official Review of Submission42557 by Reviewer 5Dwi
Dear reviewer,
Thank you for your helpful review.
[ 1. On paper summary ]
We intend to contribute by showing credit assignment can be improved for black box symbolic regression, and the multiple-testing bias can be adjusted.
We focus on black box regression in general, and the finance was used as an example. We do not claim tradable market anomaly.
We updated our method, and now our SRBench result is based on DSR, and we modified minimal amount of code to test our method. https://anonymous.4open.science/r/token-level-credit-assignment-FBE1/
[ 2. On critical weakness: ]
Thank you for letting me know about the logic and structure problem. I tried my best to completely rewrite the paper, to make it simple and easy to read.
To improve the reproducibility of our financial backtest, we added our backtest result for equations from Alphagen paper (First line of table 2, and table 5.) We refered Qlib for transaction cost implementation for our backtest.
Please note that we updated our claim, thanks to your advice. We do not claim SOTA nor tradable market anomaly or prediction performance in finance.
[ 3. Answer to the question: ]
Currently we updated the model, and we using the same RNN model with well-known DSR paper, because our previous model was too large to claim generalizability.