acceptodds
Under review as a conference paper at ICLR 2027

Knowing When to Abstain: Risk-Aware Criteria for Action-Level Selectivity in Trading Agents

Abstract

Despite continuous performance improvements of large language models, unreliable outputs may still be generated due to the inherent uncertainty of the models. For LLM-based trading agents,unreliable and useful actions may coexist within trading recommendations, which can lead to poor portfolio returns and elevated risks. One solution is to abandon those high-risk trading actions. For this purpose, Risk-Aware Action Selective Execution Criterion (RAASEC) is proposed. Three metrics are adopted to evaluate the impact of model uncertainty on each action, and two additional metrics are adopted to assess the market risk. The criterion ensures that only low-risk actions are selected for execution via an adaptive risk–coverage balancing strategy.Backtesting on the challenging China A-share market demonstrates that endowing agents with the ability to abstain from high-risk actions effectively enhances portfolio returns and outperforms the sort-of-states methods. Experiments also verify the complementarity of the five metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Confidence Estimation, Risk-Aware Decision Making, Quantitative Finance, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QTq3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes RAASEC, a filter that decides which individual actions from an LLM trading agent to execute. Each candidate action, sampled under 10 perturbation prompts, is scored by five confidence and risk measures, combined by a geometric mean, and the top fraction is executed. On a 2025 backtest of CSI 300 stocks with Qwen3-32B, RAASEC reports 52.57% annualised return and a Sharpe ratio of 2.11, outperforming TradingAgents and FinAgent.

Critical strengths

Action-level selective execution for LLM trading agents is a practically relevant problem, and the appendix documents the data pipeline and prompts in detail.

Critical weaknesses
  • Only one 11-month backtest, one market and one base model (Qwen3-32B) are used, the reported number is the "best run" with no variance or significance tests, and no transaction costs (commission, stamp duty, slippage) are modelled.

  • The paper criticises TradingAgents and QuantAgent because their risk judgment "still relies on LLM reasoning", but RAASEC's MRBC is itself an LLM-as-judge, and the final trading agent can still adjust, defer or reject actions.

  • Qwen3 was released in April 2025, so its pretraining data may overlap the early 2025 test period, but the paper did not discuss about this, while still claiming to "thoroughly eliminate look-ahead bias".

  • The results are too good to be true. A 52.57% annualised return and 2.11 Sharpe from a 32B model in a market the authors call especially hard is extraordinary, and FinAgent's active-day count implies 234 trading days (163 / 69.66%) while every other row implies 232.

  • Writing errors and contradictions:

    • Threshold direction is reversed between Sec. 4.3 and Sec. 5.2.
    • The same ablations report different numbers in Tables 3 and 4 (e.g., "w/o RFBC": 44.81% vs. 32.24%).
    • Figure 1's caption refers to a nonexistent "Confidence Filtering Agent".
    • Typos include "sort-of-states methods", "Adapative", "Assesing" and "MRPC".
Questions for authors

Please see the weakness section

Reviewer_toJUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets action-level abstention for LLM-based trading agents. It proposes Risk-Aware Action Selective Execution Criterion (RAASEC), which scores each candidate trading action with three model-uncertainty metrics (RVC, RFBC, TPBC) and two market-risk metrics (IPSMRBC, MRBC), then uses an adaptive coverage threshold to execute only sufficiently reliable actions. The framework is evaluated on CSI 300 A-share constituents from January to December 2025 using Qwen3-32B as the base model. The main claims are that action-level selective execution improves portfolio metrics over baselines (annualized return 52.57%, Sharpe 2.11, IR 1.07, alpha 36.34%) and that the five metrics are complementary, with removing IPSMRBC causing the largest degradation and removing TPBC the smallest.

Critical strengths
  1. The paper identifies a genuine and underexplored problem: action-level confidence and selective abstention for LLM trading agents, rather than whole-response uncertainty. The framework cleanly separates model-internal uncertainty from market/portfolio risk, and the ablation design directly tests the contribution of each metric.
  2. The evaluation includes multiple complementary layers: metric-level validity (AUROC/AUPRC/ECE), end-to-end trading comparison, threshold sensitivity, and per-metric ablation. This structure makes the method easier to compare against future work and provides a reasonable basis for the "action-level abstention helps" claim, assuming the reproducibility gaps are fixed.
Critical weaknesses
  1. The paper-code mismatch is material: the supplementary implementation supports score_h in the geometric mean and scaling of the coverage threshold by a global score_a * score_c term; the backtest result tool defaults to enabling both, while the shipped agent config disables score_h but still enables scale_by_scores. The paper describes only five metrics and a direct threshold fraction. Without stating which configuration produced Tables 2-4 and the ablations, the reported gains cannot be tied to the claimed five-metric RAASEC.
  2. The main financial results are not reproducible as specified: the validity labels differ between paper and shipped evaluator, transaction costs and benchmark/risk-free definitions are absent from the main text, the backtest is a single short 2025 period with no held-out robustness, and the reported "best run" has no seed distribution or confidence intervals despite T=1.0 stochastic decoding.
Questions for authors
  1. What exactly is the final aggregate used in Tables 2 and 4? Is score_h included in the geometric mean, and is the coverage fraction plain TH or TH scaled by score_a * score_c?
  2. How was Table 1 computed? Please reconcile the paper's five-day-ahead price validity definition with the execution-feasibility validity evaluator in the supplementary code.
  3. Are the reported returns net of transaction costs, fees, slippage, stamp duty, and market impact? If so, where is this specified, and what is the effect on the 52.57% AR and 2.11 SR?
  4. Please provide exact formulas and sources for AR, alpha, beta, SR, IR, and MDD, including benchmark, risk-free rate, and annualization. Why does Buy and Hold have alpha 2.96% and beta 0.3770, and why is its IR -0.42?
  5. Why is the validation set 2024-01-01 to 2024-04-01 while the test is 2025? Is this a year-lagged scheme or a typo? Please also reconcile test end dates 2025-12-16 and 2025-12-18.
  6. How many runs/seeds produced Tables 2-4? What was the distribution around the "best run"?
  7. Why does RAASEC have MDD 16.83%, higher than several baselines, when the paper frames it as risk-aware selection? Why does removing RFBC reduce MDD to 6.49%?
  8. Figure 2 says active days are minimized at TH=35%; Table 4 uses TH=30 with 92 active days. Please clarify the sensitivity curve.
  9. FinAgent reports 163 active days and 69.66%, but 163/232 = 70.26%. Is the denominator 232 or is one value a typo?
  10. What exactly do the "Only Analysis" and "Analysis+Verify" baselines execute? Do they use one candidate or the same 10 perturbation samples as RAASEC?
Reviewer_4ogpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RAASEC for selectively executing individual LLM-generated trading recommendations. Candidates generated under multiple perturbation prompts are ranked using five scores covering decision stability and market-based feasibility, then screened before execution verification. Backtests on CSI 300 constituent stocks with Qwen3-32B report 52.57% annualized return, compared with 41.05% for TradingAgents. The evaluation also includes action-level diagnostics and component ablations.

Critical strengths
  1. Selecting which individual trading actions to withhold is a useful problem. Combining consistency across model outputs with external market evidence is a reasonable approach.
  2. Action-level diagnostics and five component ablations provide evidence beyond end-to-end comparisons. In particular, removing IPSMRBC substantially reduces the reported return.
Critical weaknesses
  1. The adaptive screening rule is unclear. Sections 4.3 and 5.1 select the coverage rate TH on validation data and keep it fixed during testing. Section 5.3 instead describes confidence-driven adjustment and attributes the return difference of 52.57% versus 30.12% to adaptive versus fixed coverage. With fixed TH and candidate count O, the rule also becomes fixed Top-k, yet Table 3 reports another result for that variant. Without an update rule and precise control definitions, the contribution of this central mechanism cannot be verified.
  2. Appendix B.1.5 assigns cutoff(t) to documents whose publication times cannot be parsed and retains them under a permissive mode. This can admit documents published after the simulated decision. The paper does not establish whether the fallback affected agent inputs or whether leakage actually occurred. An audit or a backtest excluding documents without verified historical availability is needed to support the reported results.
Questions for authors
  1. What are the exact daily rules for the full method, fixed coverage, and fixed Top-k? Is TH updated during testing, and is O fixed?
  2. Was the permissive timestamp fallback enabled, and how many agent-visible documents were affected? Do the main gains persist when documents without verified historical availability are excluded?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.