Empirical step reward for RL-based Symbolic Regression

TD-based advantage estimation (For example, one step forward return $Q^{\pi }(s_t,a_t)=E_{\pi }[r_t+\gamma Q(s_{t+1},a_{t+1})]$) can be biased when symbolic regression, because the optimal policy is non-ergodic, and there is only terminal reward($Q^{\pi }(s_0,a_0)=E_{\pi }[{\gamma }^{\tau }{r}_{\tau }]$ ). For example, dataset choice can be important, and wrong symbol choice can be not recoverable in symbolic regression. On the other hand, Monte-Carlo sampling ${\gamma }^{\tau -t}{r}_t(\tau )-V(s_t)$ shows high variance which makes training difficult. Tail distribution can be used(Petersen et al., 2021) to partially mitigate this limitation. In our research, intermediate fitness of the equation is obtained for each step, which enables calculation of empirical reward for each step, defined by improvement of the fitness $\varphi (s_{t+1})-\varphi (s_t)$. Our method is based on virtual stack machine, which maps each symbol to stack state update. Our empirical step reward can be seen as finding useful heuristic segments, which is inspired by traditional genetic programming. Our method was verified using financial factor search task, with adjustment for multiple testing bias. Increased diversity of the symbolic expression set is observed in mean-variance optimized portfolio return.