acceptodds
Under review as a conference paper at ICLR 2027

Decision Confusion: Learning More Can Decide Worse under Weak Signal

Abstract

Expressive learned policies are increasingly used in sequential decision-making because they can discover complex representations and decision rules directly from experience. However, when decision-relevant predictive signals are weak relative to noise, additional flexibility may increase finite-sample estimation error without providing commensurate gains in exploitable structure. We investigate how flexibility in policy learning affects performance under weak signals. We argue that the appropriate extent of learning is governed by predictive information: under weak signals, additional flexibility induces decision confusion, making some structure better left unlearned. Specifically, we demonstrate that, in environments characterized by high noise and bounded drift, a restricted rule can outperform an expressive learner, and that increasing the sample size may not eliminate this advantage. Using a signal-to-noise ratio (SNR)-controlled Heston-style environment, we show that learning only a selector over fixed rules matches the strongest fully learned policies at low SNR, whereas end-to-end reinforcement learning catches up only at high SNR. Building on this finding, we propose Diverse-Cue Hierarchical Reinforcement Learning, a method designed for real financial markets. Experiments on eight indices from the Shanghai and Shenzhen stock exchanges show that our method attains the best total return on six of the eight markets and the best Sharpe ratio on seven. This paper offers a new perspective on architecture design under weak signals: the granularity of learning should scale with predictive information rather than model capacity.

Primary area:
reinforcement learning
Keywords:
Reinforment Learning, Fast-and-Frugal Heuristic, SNR

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_vTpSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is difficult to follow. The introduction is not clear, and neither are the following sections. It's hard to understand (1) what the problem is; (2) what the limitations of current approaches are; and (3) how the authors aim to overcome these limitations. From what I could understand, the remaining question is: why is HRL an appropriate solution for this extremely noisy setting?

I would expect the authors to provide this justification in the Related Work section at the latest. The paper mentions relevant works such as EarnHFT, but without explaining their limitations or including the complete methods as baselines for comparison. The introduction also mentions expressiveness power and overfitting, but the connection between these issues and the proposed solution is not clearly established or empirically demonstrated.

At the current stage, the presentation makes it difficult to assess the contribution. I believe this paper needs substantial revision before being submitted again.

Critical strengths

The problem seems to be interesting, although the presentation did not highlight it.

Critical weaknesses

The presentation makes it difficult to assess the contribution.

Questions for authors

Why weren't the papers in the Related Works used as baselines? What are their limitations?

Reviewer_hirhOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the setting where the noise is somehow stronger than the learnable signals, and claims that under low SNR, rule-based methods could be stronger than PPO methods. They proposed some theoretical metrics to quantify this scenario. And in some trading benchmarks, they demonstrated that rule-based methods could be empirically stronger.

Critical strengths

None

Critical weaknesses
  • Theorem 1 claims that the learned regret has an upper bound and a lower bound so that a fixed policy is better than than the learned policy if the fixed policy'regret is small than the lower bound. This conclusion is trivial with little information, while its the major theoretical result of this paper. Furthermore, the paper provides neither a truly practical estimator nor statistical guarantees for estimating these quantities.
  • The proposed decision confusion mechanism is not directly validated. They only compare empirical returns across different SNR settings, and the results are unsurprising.
  • The paper writing is very bad, requiring substantial polishment.
Questions for authors

no

Reviewer_Bwg2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This submission proposes Diverse-Cue Hierarchical Reinforcement Learning, a method designed for real financial markets. Experiments are conducted in eight indices from the Shanghai and Shenzhen stock exchange.

Critical strengths

Please see below.

Critical weaknesses

The writing of this submission is not clear at a level, almost trying to hide the actual message and contributions of the paper. The paper is not well connected, mostly semantically empty and incoherent.

At some parts, fundamental concepts like bias-variance tradeoff are described in several paragraphs with totally different obfuscating terminology without even referring to bias-variance tradeoff.

The entire paper follows the similar language where basic machine learning and statistical concepts are re-named/re-packaged to sound something new and mathematically complex to artificially inflate the novelty of the paper.

This is not acceptable and extremely misleading.

Given these issues, I do not think this paper is ready to be published.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.