- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RetailAgent, a prompting protocol in which an LLM (mainly Qwen3.5-9B) repeatedly chooses long or flat on anonymized intraday CSI-500 price histories, given as text, a chart, or both, optionally with account state or self-written memos. Decisions are scored with a within-stock, exposure-matched timing alpha. Across 14 standard configurations, the timing alpha is consistently negative (about −30 to −57 bps per stock-day). Memo-based self-conditioning is associated with more persistent positions and more negative timing. The authors interpret this as structured, wrong-signed sequential failure of LLM agents.
Critical strengths
- The research question (directional rather than noise-like failure in sequential LLM decisions) is interesting.
- The estimand is clearly defined, it is kept separate from cross-sectional IC, and confidence intervals are clustered by date.
- The authors are careful not to overclaim economic or trading implications.
Critical weaknesses
-
The authors say that replacing the random null with a price-level or last-return rule "changes the comparison dramatically," because those rules have nonzero timing of their own, but no numbers are reported. This suggests that simple momentum rules may show equally negative (or worse) timing, so the result may reflect a trend-following policy in a mean-reverting intraday market rather than a failure specific to LLMs. The timing of momentum, breakout, and intraday-trend rules over several lookback windows should be reported alongside the LLM arms.
-
I think the model coverage does not support claims about "LLM agents" in general. Most results come from models of 9B parameters or fewer. The Haiku 10-minute check uses only 165 stock-days, no frontier model is evaluated, and reasoning mode is disabled basically. The prompts are not provided. If they contain any retail-investor persona (the name suggests this is possible), the negative timing could be persona-induced. The full prompts need to be released to rule this out.
-
"All 14 configurations are negative" is presented as repeated evidence, but the configurations actually share the same model family, market, and largely overlapping periods. They are repeated measurements of one effect, not independent replications. Each configuration is also run once at temperature 0.7, with no variation across random seeds. This is important for inference-type of research as I think asking for multiple random seed for training is useless but for inference it still affects much.
-
The quality of the figures are generally not good. Please consider provide the pdf version or svg version.
Questions for authors
See the weakness section.
2 comments
Thank you for the response and the extra simple-rule results. But they somehow reinforce my main concern about the contribution. Trend-following in a mean-reverting intraday market is part of the explanation, and you confirm that the prompt explicitly assigns a retail-investor persona. Together, these suggest the main finding largely reduces to "a persona-prompted LLM behaves like a trend follower and therefore times intraday reversals poorly", which is expected rather than a new insight into LLM agents. The counter-policy perspective that would make this structure more meaningful is left to future work. I will therefore keep my score.
Also, although several changes are promised (the new table, the revised wording, and the PDF figures), I do not see any of them reflected in a revised draft.
Response to Reviewer 9LdQ
Thank you for the careful review. One clarification of motivation first, since it shapes our answers below.
We are not ranking the LLM as a forecaster. Our question is whether a sequential policy's mistakes are directionally structured, because a participant that reacts to prices in a systematic way is predictable to others, even when its forecasts are poor. Other participants could condition a response on that regularity. The present experiment supplies an empirical starting point for a future counter-policy problem. The contribution is therefore the structure of the tested policy and how its own state changes it, not the magnitude of the alpha or a claim that all LLMs fail.
Simple rules. We agree these should be reported. On the same data the LLM scores −45.7 bps per stock-day. The suggested rules give: | Rule | 10 min | 30 min | 50 min | |---|---|---|---| | Momentum | −15.6 | −17.1 | −21.1 | | Breakout | −13.6 | −14.8 | −18.7 | | Fitted trend | −14.9 | −16.2 | −20.0 | You are right that trend-following in a mean-reverting intraday market is part of the explanation: these rules are also negative. They reach less than half of the LLM's magnitude at every lookback. Consistent with this, projecting the inverted LLM schedule on our GRU, GBDT-38, and reversal signals leaves +43.9, +40.4, and +38.9 bps, with intervals above zero. The LLM policy overlaps with simple trend rules but is not reduced to them by these tests. We will add the table to the paper and revise the sentence you quoted.
Model coverage and prompts. We cover 4 families spanning Qwen/Phi/Haiku/Granite, while frontier model coverage remains open. The Haiku with 5 min horizon has 827 stock days which provides further evidence. The full templates are already supplied with our code implementations. The standard prompt does explicitly say “You are a retail investor watching one stock during the trading day.” This is also compatible with our motivation: retail investor could have certain common persona that shapes the market. Likewise, principal Qwen runs disable thinking.
Overlap and decoding seeds. We agree on both points. The 14 configurations are robustness checks on shared data, not independent replications, and we do not pool their significance. Each temperature-0.7 run keeps one trajectory per stock-day, so the date-clustered intervals do not capture seed variance; that component is unmeasured.
Figures. We will provide those pdf figures and improve label sizes and panel layout in the revision. Thank you again for your feedback!