- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes STOCK-R1, a Qwen3-based financial LLM that does not write future prices directly. Instead, it emits a structured JSON "forecast action" (direction, end-point change, volatility and drawdown bins, turning points, and so on), which is passed as a tool call to a separately pretrained time-series decoder. The decoder conditions on OHLCV history and the action and returns a 10-day trajectory, which the LLM then uses to answer the question. Training has three stages: probabilistic pretraining of the time-series module, SFT on GPT-5 distilled traces, and GRPO with rewards for answer correctness, action accuracy, trajectory precision, and action/trajectory consistency, with samples reweighted by predicted uncertainty. The model is evaluated on a self-built S&P 500 QA benchmark, on forecasting metrics against time-series and LLM baselines, with ablations, and with a backtest.
Critical strengths
[1] The action interface is a sensible design idea.
Asking the LLM to commit to a small set of checkable trajectory attributes, and then letting a dedicated decoder produce the numbers, is a cleaner division of labor than having the LLM write long numeric strings. It also gives field-level reward signals that free-form rationales do not. The action-field masking in Table 11 and the conditioned vs. unconditioned tool comparison in Table 3 point in the expected direction, and the SBUX case in Appendix C.6 shows the intended behavior concretely.
[2] The data pipeline is described with some care, and the set of time-series baselines is fairly wide.
Appendix A.1 explains how news and fundamentals are cut off before the forecast date, which is the right concern for this kind of benchmark. Table 2 also includes both zero-shot foundation models and models trained from scratch, which makes the forecasting comparison more informative than in many similar papers.
Critical weaknesses
[1] The main QA comparison is not fair, and the control for the action interface is not well documented.
Table 1 compares STOCK-R1, which was trained with SFT and RL on 40K questions from the same GPT-5 generator and answer format as the test set, against zero-shot prompting of general and financial LLMs. I think most of the headline gain comes from this. Forecast QA makes it clear: GPT-5 gets 8.16 and o3 gets 2.78, which looks like a scoring or format effect rather than a reasoning gap. Qwen3-8B also scores below Qwen3-4B, which again suggests answer-format failures. The missing baseline is the same Qwen3 backbone trained with the same SFT and GRPO on the same data, but without the decoder and action interface. Table 9 already shows that giving Qwen3-4B the unconditioned tool brings most of the gain, so the remaining gap mixes in-distribution training with the action mechanism.
The evidence for the mechanism itself (Tables 3 and 10) is weakened by Appendix D, which says the controls lack seeds and confidence intervals and that the records do not establish whether the unconditioned arm was retrained or simply had conditioning removed at inference. In the second case, the drop is a distribution shift, not evidence for the interface. I appreciate the honesty, but the central ablation needs a clearly defined control. Table 1 also has no majority-class or base-rate reference, even though most items are binary questions about future returns or volatility in a test window that went mostly up.
[2] Several numbers are inconsistent, and some claims do not match the paper's own figures.
Table 1 gives STOCK-R1-4B News Analysis 61.83, but Figure 3a and Table 6 (which match every other cell of that row) give 51.83. Ten points is not a rounding issue. The 67.5 for u-GRPO in Table 4 matches neither model in Table 1, and the differences between GRPO variants in Table 4 are within about one standard deviation with no seed count, so the uncertainty reweighting claim rests mainly on an entropy curve. Section 4.3 says time-series tokens give consistent gains for all categories, but Figure 3a shows several categories where the model without them is better.
The backtest in Table 7 has impossible combinations of Sharpe ratio and return (for example, positive Sharpe with negative return for ARIMA, and the reverse for DLinear). There is also no buy-and-hold benchmark and no transaction costs, and STOCK-R1 has the highest beta in a rising market, so part of its return is just market exposure. Thus, I cannot take the backtest as evidence of trading value.
[Smaller issues]
In Table 2, STOCK-R1 is not the best on several metrics and its directional accuracy is almost tied with plain Qwen3-8B, and there are no naive baselines or significance tests. The SFT teacher sees the realized future action, which may teach post-hoc rationalization. The paper does not say whether test questions are temporally disjoint from training questions. Notation for reward weights clashes with other symbols, and some details in Table 5 and the baseline configurations are missing.
Questions for authors
Q1) How exactly was the unconditioned-tool arm in Tables 3 and 10 produced? Was it retrained with SFT and RL without actions, or is it the same model with conditioning removed at inference? If it is the latter, a rerun with matched training and a few seeds would be needed before the comparison says anything about the action interface.
Q2) Could you report Qwen3-4B and 8B trained with the same SFT and GRPO on the same 40K questions and context, but with no decoder and no action, together with majority-class and always-up references for each QA category? This is the comparison that would tell me how much of the gain comes from the proposed mechanism, and it would change my assessment.
Q3) Which value is correct for STOCK-R1-4B News Analysis, 61.83 or 51.83, and which model produces the 67.5 in Table 4? Could you also explain the Sharpe and return signs in Table 7?
Q4) Is the QA train/test split temporal, with all test questions dated after every training question? Also, can the policy game the consistency reward by emitting actions that the frozen decoder reproduces easily, such as near-flat profiles? The distribution of emitted directions and volatility bins before and after RL would help to check this.