acceptodds
Under review as a conference paper at ICLR 2027

RetailAgent: Auditing Systematic Timing Errors in Self-Conditioned LLM Agents

Abstract

Sequential LLM agents can fail in ways that are directional rather than noise-like. We introduce RetailAgent, a controlled multimodal, state-conditioned framework that repeatedly chooses long or flat from an anonymized intraday price history, its chart representation, and permitted account state. RetailAgent provides a sparse price-and-state interface for auditing sequential decisions; it neither represents a retail population nor simulates a full market. We evaluate each decision trajectory using exposure-matched within-stock timing, which tests alignment with subsequently revealed return labels rather than executable portfolio performance. Across text, chart, joint text–chart, and account-state conditions, all estimates are negative; the principal 10-minute text, chart, and multimodal arms are , , and bps per stock-day. This wrong-signed timing is directional: the complementary schedule reverses its score, while linear projection on conventional price models leaves substantial residual timing. Narrative self-conditioning further reorganizes the trajectory: stronger wrong-signed timing coincides with fewer position changes in the tested memo conditions. We interpret these findings as structured sequential failure and as motivation for future counter-policy and leader–follower analyses, not as evidence of profitable trading, market impact, or human retail equivalence.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM agents, sequential decision-making, financial markets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9LdQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RetailAgent, a prompting protocol in which an LLM (mainly Qwen3.5-9B) repeatedly chooses long or flat on anonymized intraday CSI-500 price histories, given as text, a chart, or both, optionally with account state or self-written memos. Decisions are scored with a within-stock, exposure-matched timing alpha. Across 14 standard configurations, the timing alpha is consistently negative (about −30 to −57 bps per stock-day). Memo-based self-conditioning is associated with more persistent positions and more negative timing. The authors interpret this as structured, wrong-signed sequential failure of LLM agents.

Critical strengths
  • The research question (directional rather than noise-like failure in sequential LLM decisions) is interesting.
  • The estimand is clearly defined, it is kept separate from cross-sectional IC, and confidence intervals are clustered by date.
  • The authors are careful not to overclaim economic or trading implications.
Critical weaknesses
  • The authors say that replacing the random null with a price-level or last-return rule "changes the comparison dramatically," because those rules have nonzero timing of their own, but no numbers are reported. This suggests that simple momentum rules may show equally negative (or worse) timing, so the result may reflect a trend-following policy in a mean-reverting intraday market rather than a failure specific to LLMs. The timing of momentum, breakout, and intraday-trend rules over several lookback windows should be reported alongside the LLM arms.

  • I think the model coverage does not support claims about "LLM agents" in general. Most results come from models of 9B parameters or fewer. The Haiku 10-minute check uses only 165 stock-days, no frontier model is evaluated, and reasoning mode is disabled basically. The prompts are not provided. If they contain any retail-investor persona (the name suggests this is possible), the negative timing could be persona-induced. The full prompts need to be released to rule this out.

  • "All 14 configurations are negative" is presented as repeated evidence, but the configurations actually share the same model family, market, and largely overlapping periods. They are repeated measurements of one effect, not independent replications. Each configuration is also run once at temperature 0.7, with no variation across random seeds. This is important for inference-type of research as I think asking for multiple random seed for training is useless but for inference it still affects much.

  • The quality of the figures are generally not good. Please consider provide the pdf version or svg version.

Questions for authors

See the weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9LdQ

Thank you for the careful review. One clarification of motivation first, since it shapes our answers below.

We are not ranking the LLM as a forecaster. Our question is whether a sequential policy's mistakes are directionally structured, because a participant that reacts to prices in a systematic way is predictable to others, even when its forecasts are poor. Other participants could condition a response on that regularity. The present experiment supplies an empirical starting point for a future counter-policy problem. The contribution is therefore the structure of the tested policy and how its own state changes it, not the magnitude of the alpha or a claim that all LLMs fail.

  1. Simple rules. We agree these should be reported. On the same data the LLM scores −45.7 bps per stock-day. The suggested rules give: | Rule | 10 min | 30 min | 50 min | |---|---|---|---| | Momentum | −15.6 | −17.1 | −21.1 | | Breakout | −13.6 | −14.8 | −18.7 | | Fitted trend | −14.9 | −16.2 | −20.0 | You are right that trend-following in a mean-reverting intraday market is part of the explanation: these rules are also negative. They reach less than half of the LLM's magnitude at every lookback. Consistent with this, projecting the inverted LLM schedule on our GRU, GBDT-38, and reversal signals leaves +43.9, +40.4, and +38.9 bps, with intervals above zero. The LLM policy overlaps with simple trend rules but is not reduced to them by these tests. We will add the table to the paper and revise the sentence you quoted.

  2. Model coverage and prompts. We cover 4 families spanning Qwen/Phi/Haiku/Granite, while frontier model coverage remains open. The Haiku with 5 min horizon has 827 stock days which provides further evidence. The full templates are already supplied with our code implementations. The standard prompt does explicitly say “You are a retail investor watching one stock during the trading day.” This is also compatible with our motivation: retail investor could have certain common persona that shapes the market. Likewise, principal Qwen runs disable thinking.

  3. Overlap and decoding seeds. We agree on both points. The 14 configurations are robustness checks on shared data, not independent replications, and we do not pool their significance. Each temperature-0.7 run keeps one trajectory per stock-day, so the date-clustered intervals do not capture seed variance; that component is unmeasured.

  4. Figures. We will provide those pdf figures and improve label sizes and panel layout in the revision. Thank you again for your feedback!

Thank you for the response and the extra simple-rule results. But they somehow reinforce my main concern about the contribution. Trend-following in a mean-reverting intraday market is part of the explanation, and you confirm that the prompt explicitly assigns a retail-investor persona. Together, these suggest the main finding largely reduces to "a persona-prompted LLM behaves like a trend follower and therefore times intraday reversals poorly", which is expected rather than a new insight into LLM agents. The counter-policy perspective that would make this structure more meaningful is left to future work. I will therefore keep my score.

Also, although several changes are promised (the new table, the revised wording, and the PDF figures), I do not see any of them reflected in a revised draft.

Reviewer_KpjDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RetailAgent studies binary long/flat LLM decisions from anonymized intraday price histories under text, chart, account-state, narrative-memory, and optional learned-conditioning settings. Its main diagnostic is within-stock timing relative to the agent's average exposure, which is distinguished from portfolio return and cross-sectional stock selection. The tested standard configurations have negative timing estimates, while narrative memo conditions alter timing and persistence. The paper supplements these observations with conventional-signal projections, trajectory shuffling, model-family checks, and a separately trained LoRA stress test, while limiting claims about human retail behavior and executable trading performance.

Critical strengths
  1. A clearly defined behavioral diagnostic with relevant controls. Equations 1–2 distinguish within-stock timing from cross-sectional selection. The exposure-matched random null, conventional-signal projections, and Appendix F shuffling make the negative timing observations more interpretable. In particular, shuffling attenuates the principal text estimate from −45.7 to −3.5 or −8.7 bps, supporting the importance of the original action–return alignment within the studied data.
  2. Useful breadth of descriptive evidence with appropriate uncertainty for the main estimates. Table A2 reports negative timing across 14 standard configurations and uses date-clustered intervals; Appendix E adds model-family checks. The memo sweep also records timing, IC, and persistence, providing a useful basis for further controlled analysis. The paper explicitly limits claims about human retail behavior, executable profits, algebraic inversion, and causal mediation. These are valuable observations, although the different arms are not independent replications.
Critical weaknesses
  1. The incremental memo-history effect needs a more direct comparison. Sections 3.3 and 4.1 explicitly state that w=0 already generates and reads a current memo; it is not a no-memo baseline. Standard actions use temperature 0.7, whereas second-pass memo actions are greedy, so a standard-versus-memo contrast also changes the inference procedure. The more focused w=1 versus w=0 comparison shares that memo procedure, but its −11.3 bps mean difference has no paired date-clustered interval in Table A2. Separate intervals do not establish the uncertainty of the contrast. The persistence association is informative but, as acknowledged, does not identify mediation. The current evidence therefore supports a descriptive association more directly than a well-isolated self-conditioning mechanism.
  2. Differential failure handling and sample composition leave a selection concern. Section 4.1 maps standard-arm parse failures to flat but excludes failed memo decisions from scoring. Failure rates and the precise exclusion unit are not given, so it is unclear whether missing decisions alter the exposure mean or evaluated return intervals in Equation 1. Table A2 also uses 13,710/14,937/14,438 stock-days for the main text/chart/multimodal arms. The authors appropriately avoid ranking unmatched magnitudes, but comparable information sets do not by themselves ensure comparable evaluated populations. Without failure counts and a common-rule sensitivity check, the robustness of cross-condition comparisons to this selection remains uncertain.
  3. The temporal and cross-model checks are not documented sufficiently to establish an independent confirmation. Table A4 supplies later-period point estimates, but no corresponding intervals, sample sizes, calendar boundaries, or clear ordering of prompt selection relative to that evaluation. Section 4.1 specifies a relative chronological split for the learned encoder, rather than a complete calendar account of the inference-only experiments. Appendix E's sign checks are useful, but some have small samples or unavailable intervals. Appendix G's 60 held-out dates concern an explicitly adverse-trained LoRA and do not confirm the inference-only memo effect. Source-panel dates and selection rules, together with model and prompt version records, are needed to assess the reach and reproducibility of the stability claim.
  4. The latent branch does not yet answer its stated comparison with narrative conditioning. Section 3.3 motivates comparing linguistic self-commitment with additional latent state, but Appendix B states that different prompts, samples, and training procedures preclude a causal comparison with memo-only arms. Table A6 also mixes cross-sectionally centered and other transformed objectives without specifying the latter transformations there. These are acknowledged differences, yet the main scientific question remains unresolved. A related ambiguity appears in Section 4.4: encoder-word timing of −51.5 is described as worse than −70.2, although it is less negative under Equation 1. The manuscript needs to specify which objective is being judged and what existing matched evidence supports the cross-channel interpretation.
Questions for authors
  1. Memo attribution: Can the saved matched trajectories provide a paired date-clustered interval for w=1 minus w=0? Is there an existing no-memo or equal-procedure neutral-note control on the same stock-days with the same decoding settings? Please state exactly which conclusion concerns earlier memo history and which concerns the whole two-pass procedure.
  2. Selection and parsing: What are the failure counts and rates for each principal arm, and are failed memo decisions removed at the interval or stock-day level? How are exposure and subsequent state calculated after a failure? Please report a feasible sensitivity comparison using common eligibility and a common failure rule on existing records.
  3. Temporal confirmation: Please specify the data source, calendar ranges, stock-day selection, and sample sizes for the main and Table A4 later-period runs, together with whether prompts were fixed before the latter results were inspected. Can existing records supply date-clustered uncertainty and model/prompt version identifiers, and identify any untouched inference-only evaluation separately from the trained LoRA experiment?
  4. Objectives and interpretation: Please give the objective transformations used in Table A6 and identify any existing matched comparison that distinguishes narrative commitment from latent-state conditioning. Also clarify the intended direction of “worse” for −51.5 versus −70.2. If the branches support only separate descriptive findings, which narrower claims should the main text make?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KpjD

Thank you for recognizing the distinction between within-stock timing, cross-sectional selection, and executable portfolio performance, and for identifying specific comparisons that would make the evidence clearer.

We first clarify the broader motivation. In a market with adaptive participants, the decision rule of another participant can itself be useful information. A participant need not repeat identical trades for this to matter: recurring responses to salient price movements, recent gains or losses, and prior commitments may make its conditional behavior predictable. A counterparty could seek to anticipate or respond to those regularities. This motivates studying whether a sequential policy has systematic directional structure, including structure in its mistakes, rather than assessing it only by its success as a return forecaster. It also motivates our interest in whether self-generated state makes that behavior more persistent. We hope this helps explain why structured adverse timing and its sensitivity to conditioning are meaningful objects of study, even before an adaptive counter-agent is introduced.

1. Memo-history attribution

The appropriate history comparison is w=1 versus w=0, which shares the two-pass procedure and decoding settings; w=0 already reads the current memo. Removing the union of failed intervals, the paired difference is −8.24 bps [−15.36, −1.45] with nonconstant schedules in both arms. Thus the history effect is supported among active trajectories. The evidence therefore supports sensitivity to carried narrative state, without establishing that the exact identity of one's own preceding note is essential or that persistence causally mediates the effect.

2. Parsing and sample composition

Recorded parse failures are 0-4%. Failed memo intervals and their returns are excluded, and exposure is recomputed over retained intervals. Generation retains the previous position on a failure, so dropping that interval does not undo subsequent state effects. Restricting the paired history comparison to entirely parse-clean, nonconstant stock-days still gives −16.03 [−29.04, −2.80].

On a common sample of 14,986 complete, parse-clean stock-days, including constant schedules, text, chart, and joint inputs score −41.76, −29.79, and −47.08 bps, respectively, with all date-cluster intervals below zero. Adverse timing thus persists under common eligibility.

3. Temporal confirmation

The source universe comprises stocks in CSI-500 during 2024–2025 (membership snapshot at beginning).

4. Latent objectives and interpretation

Table A6 uses AR(1)-residualized returns for F/G/H, cross-sectionally demeaned returns for Passive/I/K8/K16/Vocab-500, and AR(1) residualization followed by cross-sectional demeaning for J. These are different objectives and cannot support one numerical capacity ranking.

“Worse” at −51.5 versus −70.2 meant weaker amplification of adverse timing. The clearer wording is “encoder-word injection attenuates negative timing”; under Equation 1 it improves the agent's timing. The paired word-injection-minus-off difference is +24.63 [15.93, 33.42] bps on 810 nonconstant pairs. This concerns literal word injection, not a matched test of narrative versus embedding-space conditioning. The branches support separate descriptive findings, without identifying a causal cross-channel ranking.

I thank the author for the rebuttal that partially resolved my concerns and will retain my rating.

Reviewer_836KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper finds that when LLMs predict short-term returns based on text and charts, they make systematic errors. Reversing these errors yields a positive Timing Alpha. When LLMs are exposed to their previous judgments, their wrong decisions become strengthened.

Critical strengths

[1] The paper provides an interesting framework to evaluate LLMs’ decision failures in terms of systematic behavioral errors.

[2] I find it interesting that LLMs’ previous explanations can generate self-conditioning.

Critical weaknesses

[1] The paper’s decision-making interval is 10 minutes. Although I understand the authors’ intention in choosing short windows, I would be interested in seeing the results when the investment window is 12 hours, 1 day, or even longer. If Timing Alpha turns from negative to positive over longer horizons, the paper’s behavioral implications will be substantially different. If LLMs make the same mistakes over longer horizons, the findings will strengthen the authors’ current conclusion.

[2] Relatedly, I am not sure whether LLMs are well-suited for making short-horizon investment decisions. LLMs are originally designed to perform text-related tasks. Short-horizon price movements, however, are generally more statistical, not context-based (or fundamental). Therefore, I am not very convinced by the authors’ implication that RetailAgent simulates behavioral trading tendencies.

[3] The authors motivate their study by citing papers like Barber and Odean (2000, 2008), which explore retail investors’ behaviors. I see that the authors acknowledge that RetailAgent does not represent an average retail investor. However, if the authors wish to establish LLMs’ behavioral tendencies, it is imperative that the authors conduct human subject surveys and compare the results with the LLMs’ outputs. By doing so, the authors can conclude whether the documented systematic failure of LLMs is common among human investors or not.

[4] I also encourage the authors to understand ‘why’ LLMs are making biased decisions. If these decisions are systematic, it’s worth exploring and understanding what causes these errors. Are they due to LLMs being fixated on simple price movements or to short-term momentum? If the authors can identify the main cause of the errors, they can adjust the prompt and examine whether Timing Alpha improves.

[5] Unless I am mistaken, the paper does not disclose exactly how many firms are used in the experiment, whether all 500 firms in the CSI-500 are included, or the exact trading days. If a subsample is used, the authors should disclose the selection criteria. Also, it seems like choosing diverse dates matters a lot for this study, as different market regimes can make the LLMs’ predictions mechanically better or worse.

[6] What is the date range of the CSI-500 data? If the trading days overlap with the LLMs’ training window, this test is inherently contaminated by lookahead bias.

Questions for authors

Please respond to my comments in the Weakness section.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 836K: Motivation, Horizon Choice, and Scope of Claims

We thank the reviewer for the thoughtful questions. We address [1] and [2] together, because we think the second point explains our choice in the first.

[1–2] Horizons and task suitability. We agree that short-horizon price movement is technical/statistical rather than contextual, and that LLMs are not designed to forecast it. Our subject is a decision-maker that reacts to price patterns without a statistical edge, and our question is whether its errors are noise-like or directionally structured. The experiment tries to take advantage of the similarity between LLM and human decision making, instead of evaluating their ability on short-term forecasting. The intraday setting is chosen for that reason. Within a session, the trading information is nearly the only input, so the policy’s reaction can be isolated. Meanwhile (also as the reviewer points out) longer horizon has more to do with information digest and fundamental research, which will disrupt the subject we focus. A separate study on that topic would be informative, but about a different information set. We also evaluate 5 and 20 minute intervals, respectively at -38.8 and -56.7bps, they show no movement toward zero as the interval lengthens.

[3] Relationship to human investors. We agree that human comparisons are necessary to establish whether the documented behavior is shared by actual investors, and we do not have it.

[4] Why the errors arise. This is our underlying motivation. If many participants respond to price patterns in a similar way (not necessarily mean they make same decisions), for example human traders could be easier to adapt but harder to maintain consistency and discipline, despite the exact outcome of those can be opposite in terms of buy and sell, then, better-informed participants have an incentive to trade against that response, and prices come to reflect it. The wrong sign is then a property of the policy and the market together, not of the model alone. Our data cannot test this, and we will label it as a hypothesis in the introduction. From a technical perspective, momentum reactions are a plausible contributor: simple trend responsive rules also exhibit negative timing. However, the tested momentum signals do not explain the entire LLM schedule and is not as strong as the LLM policy. We did not tune the prompt persona, as it could lead to a different policy, but our interest is in the default reaction that another participant could learn. Whether prompt changes remove the effect remains open.

[5–6] Universe, dates, and contamination. The source universe comprises stocks in CSI-500 during 2024–2025 (membership snapshot at beginning). Stock identity, calendar date, and absolute price are anonymized/withheld; only normalized price histories and permitted account state are supplied. The period likely overlaps the models’ training data, but memorized outcomes could also push timing toward positive values, which is the opposite of what we observe.

Reviewer_836KOpenReview ↗

Acknowledgement

Thank you for the response.

However, my main concerns remain unresolved. The paper’s main intended contribution requires a comparison with actual human responses, which is still missing. The paper also lacks an examination of the underlying causes of LLM errors. At a minimum, I expected an analysis of the circumstances under which an LLM tends to be more or less accurate, but the authors have not provided one.

Therefore, I retain my score.

Comments on the paper

1 comment

Dear Reviewers, With the rebuttal period ending on November 18, please take a moment to check the authors' latest responses, post your feedback, and join the discussion. We greatly appreciate your hard work and dedication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.