acceptodds
Under review as a conference paper at ICLR 2027

FinRAIN: Reliability-Aware News Ingestion for Sentiment Analysis

Abstract

LLM-based trading systems often treat financial news ingestion as a sentiment preprocessing task: each article is assigned a directional label, linked to the tickers it mentions, and passed to a downstream trading framework. This design conflates two questions: whether an article's market impact is confined to a single ticker or extends to multiple tickers, and whether the article is sufficiently reliable to support a trading signal. We propose FinRAIN, a model-agnostic news ingestion framework that transforms raw financial news into structured, reliability-aware inputs for LLM-based trading systems, with optional retrieval augmentation from external knowledge sources. FinRAIN produces article-level annotations for Market Relevance, Source Attribution, Information Novelty, Headline Exaggeration, and Stakeholder Bias. Rather than replacing existing trading agents, FinRAIN serves as a modular news processing infrastructure that can power a broader ecosystem of LLM-based trading systems by supplying higher-quality and more interpretable inputs to diverse downstream trading frameworks. Across intrinsic evaluations and downstream trading simulations, FinRAIN improves the quality of sentiment signals and achieves higher risk-adjusted returns across diverse trading settings. These results suggest that reliability-aware ingestion is a reusable interface for making financial-news pipelines more robust, interpretable, and transferable across trading frameworks.

Primary area:
interpretability and explainable AI
Keywords:
financial NLP, news ingestion, sentiment analysis, retrieval-augmented generation, algorithmic trading

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Sf87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FinRAIN, a reliability-aware news ingestion framework for LLM-based financial sentiment analysis and trading systems. FinRAIN assesses financial news along five dimensions: Market Relevance, Source Attribution, Information Novelty, Headline Exaggeration, and Stakeholder Bias. These assessments can be augmented with external knowledge through either pipeline-based or agentic retrieval. The paper evaluates FinRAIN from two perspectives. First, it examines whether reliability-weighted sentiment signals produced by five sentiment models achieve better alignment with subsequent stock returns. Second, it incorporates reliability annotations into single-stage and two-stage LLM-based trading frameworks and evaluates their trading performance. Experiments on financial news covering 20 U.S. stocks report improvements in sentiment-return correlation, directional hit rate, and risk-adjusted trading returns under several configurations.

Critical strengths
  1. Practical motivation and modular design. The paper addresses a relevant problem in financial news processing: conventional sentiment-based pipelines may treat articles with different levels of relevance, credibility, and informational value equally. The proposed five-dimensional schema provides an interpretable interface for assessing these properties and can be integrated into different downstream systems without requiring modifications to their underlying models.

  2. Evaluation across multiple models and downstream settings. The evaluation covers five sentiment analysis models, different external knowledge retrieval configurations, and two trading frameworks. The inclusion of dimension-level ablations, human agreement measurements, and computational cost analysis provides useful empirical insights into the proposed framework.

Critical weaknesses
  1. The mechanism connecting reliability scores, sentiment improvement, and trading performance is insufficiently validated. Although FinRAIN claims that five-dimensional reliability scores improve sentiment signals, the paper does not clearly describe the mathematical formulation and weighting procedure used to aggregate these scores. Moreover, the paper mainly reports aggregate metrics (e.g., sentiment-return correlation and hit rate). Although Appendix H provides a single-day case study illustrating article-level label flips and decision changes, the paper lacks a systematic analysis of how reliability weighting alters individual article contributions, sentiment directions, or prediction corrections across the full evaluation. More importantly, the sentiment evaluation (Table 2) and trading evaluation (Table 3) follow two separate pipelines. The paper does not directly verify whether improved reliability-weighted sentiment signals lead to better trading decisions. An end-to-end evaluation comparing original sentiment signals and FinRAIN-enhanced sentiment signals in the same trading framework would better support the claimed benefits. In the trading setting, FinRAIN annotations appear to be consumed as additional prompt context by the decision LLM, rather than through an explicit, reproducible weighting operator; this further obscures how the five scores mechanistically affect BUY/SELL/HOLD decisions.

  2. The robustness of LLM-based components is insufficiently evaluated. FinRAIN relies on LLM-based reliability scoring, retrieval, and trading decisions, but the paper does not provide repeated-run stability analysis. Although some experiments use deterministic decoding, the consistency of reliability scores, retrieval decisions, and BUY/SELL/HOLD predictions is not reported. Additionally, the paper does not provide variance estimates of trading metrics (e.g., Sharpe ratio and PnL) across repeated runs, making it difficult to assess whether the reported improvements are robust or sensitive to individual LLM generations.

Questions for authors

Q1: Could the authors clarify the exact formulation used to aggregate the five reliability dimensions into the final reliability weight, and explain how this weighting mechanism modifies the sentiment signals?

Q2: Could the authors separately clarify (i) how the five scores are converted into article-level weights in the intrinsic sentiment evaluation (Table 2), and (ii) how the same annotations are consumed in the trading frameworks (Table 3)—for example, via explicit weighting versus prompt-level injection? Beyond the single-day case study in Appendix H, a more systematic analysis of article-level contribution changes would help establish the mechanism.

Q3: Since FinRAIN relies on LLM-based decision-making components, could the authors evaluate whether the final trading decisions are stable across repeated runs with identical inputs?

Q4: Do BUY/SELL/HOLD decisions remain consistent across multiple executions, and what is the decision flip rate under repeated runs?

2 comments

We thank the reviewer for the constructive feedback and for recognizing the practical motivation, modular design, and broad evaluation of FinRAIN. We appreciate the opportunity to clarify how the reliability annotations are incorporated into different downstream applications.

Q1 and Q2: Reliability weighting and integration with downstream trading frameworks

An important design principle of FinRAIN is to decouple reliability assessment from downstream consumption, allowing the same structured reliability information to be utilized by different models and decision frameworks without prescribing a specific integration mechanism.

In the intrinsic sentiment evaluation, the five reliability scores are normalized to [0, 1] and combined using their geometric mean to obtain an article-level reliability weight:

Ri=(∏j=1ksi,j5)1/k,k=5.R_i = \left(\prod_{j=1}^{k}\frac{s_{i,j}}{5}\right)^{1/k}, \qquad k=5. where si,js_{i,j} denotes the reliability score for article ii and dimension jj. This formulation gives each dimension equal importance while allowing lower scores to reduce an article's contribution. The geometric mean maintains a consistent weight scale and avoids completely excluding articles with lower reliability scores.

In contrast, the downstream trading evaluation does not consume the weighted sentiment signals from Table 2. Instead, the five reliability scores, together with their corresponding rubric definitions, are directly incorporated into the trading LLM prompts alongside the original news articles. The trading LLM interprets these annotations and makes directional decisions without relying on a predefined numerical weighting operator or intermediate sentiment aggregation.

The two evaluations are intentionally designed to assess complementary applications of FinRAIN, rather than sequential stages of a single pipeline. Table 2 examines whether explicit reliability weighting improves sentiment alignment, while Table 3 evaluates whether the reliability assessments themselves can provide useful information directly to downstream trading models.

Q3 and Q4: Trading Decision Stability

We thank the reviewer for raising the question of decision stability. To address this concern, we conducted three runs for each baseline and pipeline retrieval FinRAIN configuration using identical inputs. Decision disagreement and direct BUY/SELL reversal rates were calculated from all three pairwise comparisons between runs.

ConfigurationDecision DisagreementBUY/SELL Reversals
Single-stage Baseline11%<1%
Single-stage FinRAIN6%<1%
Two-stage Baseline28–30%2%
Two-stage FinRAIN30–32%1%

The results show that FinRAIN reduces decision disagreement in the single-stage framework (11% to 6%), while the two-stage framework exhibits comparable overall variability across configurations. Importantly, direct BUY↔SELL reversals remain rare across all settings (below 1% to 2%), suggesting that most decision variability does not involve a complete reversal of trading direction.

We appreciate the reviewer's suggestion, which provides a useful additional perspective on the reproducibility of downstream trading decisions.

The authors have clarified how the five reliability scores are aggregated into an article-level reliability weight and explained the different integration mechanisms used in Table 2 (explicit weighting) and Table 3 (prompt injection). In addition, they have conducted repeated-run experiments to assess the stability of trading decisions. However, several issues remain unresolved. The authors have not sufficiently explained how the computed reliability weights modify individual article-level sentiment scores and contribute to the final aggregated sentiment signal. Furthermore, the requested systematic analysis of article-level contribution changes is still missing. Regarding decision stability, although additional experiments have been conducted, the two-stage framework still exhibits a relatively high decision disagreement rate (30–32%), with no improvement over the baseline. Therefore, I maintain my original rating.

Reviewer_6UUGOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper point out that many LLM-based trading pipelines treat news as simple sentiment preprocessing and do not check article relevance and reliability before sentiment extraction. To solve this problem, the author propose FinRAIN, a model-agnostic plug-in news ingestion framework, which uses a rubric to score each article along several reliability dimensions and optionally incorporates external knowledge under point-in-time constraint through pipeline retrieval or agentic retrieval. The paper evaluate the framework from both intrinsic sentiment aggregation and downstream trading perspectives, and reports that the proposed annotations can improve performance over baseline and that removing any scoring dimension hurts the metrics.

Critical strengths
  1. The controlled comparison design and cross-backbone consistency are good. The baseline and FinRAIN use the same inputs, and FinRAIN only adds annotations; they use the same price simulator and the same cost setting. Five backbones cover both financial-specific encoder and general LLM. The paper reports that the improvement directions are consistent under all five backbones and all trading conventions. The ablation is also repeated under FinBERT and Claude Sonnet 4.5. The direction consistency is a useful signal. But these configurations share the same news sample and the same reliability scores, so they are not independent repetitions and cannot replace interval estimation.
  2. The paper has awareness to prevent look-ahead leakage. The external knowledge sources are built offline; the retrieval is constrained by article timestamp; the release times of the five sentiment models are externally checked and are all earlier than the sample window. This is a part often ignored in financial LLM experiments. The shortage is that this prevention does not cover the knowledge in the parameters of the scoring model and the strategy LLM.
Critical weaknesses
  1. The core empirical claims lack any uncertainty quantification, and the reported effect sizes are of the same order as sampling error. The main text does not report significance tests, confidence intervals, repeated runs, or random seeds. The sample window is 132 calendar days, which by usual NYSE holidays is about 90 trading days. Under an IID approximation, with about 89 daily returns, the standard error of a single annualized Sharpe ratio is about 1.7, while the Sharpe improvements that can be computed from the main text are 0.46 (one-stage round-trip), 0.79 (one-stage long-only), 1.08 (two-stage long-short), and 1.26 (one-stage long-short), all smaller than this order. For the stock-day Spearman correlation, the null standard error is about 0.024 and can reach about 0.1 when cross-sectional correlation is strong, while the improvement at the interval endpoints without external sources is only 0.017–0.031. The hit-rate improvement is 0.9–2.4 percentage points, with a standard error of about 1.2 percentage points.2. The network-level contribution and the conclusion “topology almost has no protective effect” in the main text are not separated from the design effect.

  2. The paper lacks simple baselines that can attribute the gain to the five-dimensional reliability scheme, and the ablation only does one-by-one removal. The external comparison only has two groups: with and without FinRAIN weighting. The paper also says that market relevance scores are mostly concentrated in low bins, which indicates that many retrieved articles have limited relation to the target tickers. The ablation only removes one dimension at a time under the best agent configuration. On a corpus containing many irrelevant articles, simply down-weighting irrelevant articles may already recover most of the gain. Simple baselines such as relevance-only filtering, novelty-only filtering, or source-reliability-only weighting are missing. This mechanism also has precedents: LLT keeps only news with relevance = 100 and novelty > 90 days; a RavenPack study reports that Sharpe rises from 1.03 to 1.43 after novelty filtering (vendor-reported, not peer-reviewed); novelty conditioning can strengthen sentiment effects (working paper); source reliability weighting can improve the consistency between sentiment and next-day returns (CODS 2025). One-by-one removal at most shows that each dimension has a marginal role, and the paper does not give uncertainty of the drops, so it is insufficient to support the stronger statement that the full multi-dimensional schema is better than any simplified variant.

  3. The novelty claim is too broad and is directly contradicted by prior work that this paper itself cites and follows; the difference between the one-stage baseline and the original work is also not explained. The paper states that prior LLM-based trading research has not combined article-level materiality and reliability in evaluation. But LLT, which this paper cites and follows, already uses RavenPack relevance = 100 and event similarity days > 90 as inclusion conditions, removes near-duplicate titles, and compares third-party reports with press releases. CausalStock already lets an LLM score relevance, importance, and impact for each news × stock pair. TrustTrade already names “uniform trust” as a problem and conducts a controlled backtest; since it is only arXiv, I do not use it as a rejection reason. The more serious issue is the one-stage baseline: it follows Lopez-Lira and Tang (2023), and the baseline rows only contain article and ticker, while the corpus is deduplicated only by URL and title. If the baseline removes LLT’s original relevance and novelty filtering, then part of FinRAIN’s gain may simply be adding back filters that the original work already had, in a soft-weighted form. In this case the baseline may be weaker than the original, and the comparison cannot support the broad novelty claim. The paper should either narrow the claim or explain clearly how the baseline differs from the original and why this difference is fair.

  4. The construct-validity evidence for the reliability dimensions is weak, and the explanation that the gain comes from reliability is not yet established. The human comparison only has 100 articles, and the paper does not explain the number of annotators, their backgrounds, or inter-annotator agreement. The weighted kappa for novelty and headline exaggeration is only 0.37 and 0.36. The text says that MAD is less than 1 “among four dimensions,” and the fifth dimension is not named. FinRAIN scores are overall higher than human scores. Yahoo Finance contributes 40% of the articles, while the example for source attribution is exactly that Yahoo Finance is usually more credible than anonymous blog posts; publisher metadata maps domains to credibility tiers. By domain stratification, reposted content from different sources on an aggregator platform will be put into the same tier, so the source attribution dimension may have very low discriminative power on this 40% corpus. A risk reference (post-cutoff, concurrent, or ICLR 2026 submission; only as risk reference, not prior work) points out that surface source cues can be faked and that LLMs also have brand-level source preferences. §5.2 attributes the main effectiveness to “structured reliability-aware weighting”; before construct validity is established, this attribution may actually reflect only relevance filtering.

Questions for authors
  1. How are the weights derived? How are the five-dimensional scores combined to form these weights? Were the weighting functions, normalization methods, thresholds, and also the ±0.10 single-stage trading threshold fixed a priori (determined based on data prior to December 26, 2025), or were they adjusted during the evaluation period? The paper should clarify this point, because if these choices were tuned on the evaluation period, the reported gains may be overstated.
1 comment

We thank the reviewer for the thoughtful and detailed assessment, particularly for recognizing the controlled experimental setup and the consistent improvements across different sentiment models. We also appreciate the discussion of relevant prior research, which provides useful context for positioning our contribution.

1. Weighting formulation

Thank you for highlighting the need to clarify the weighting mechanism. The five reliability scores are normalized to [0, 1] and combined using their geometric mean to obtain an article-level reliability weight: Ri=(∏j=1ksi,j5)1/k,k=5.R_i = \left(\prod_{j=1}^{k}\frac{s_{i,j}}{5}\right)^{1/k}, \qquad k=5. where si,js_{i,j} denotes the reliability score for article ii and dimension jj. This formulation gives each dimension equal importance while allowing lower scores to reduce an article's contribution. The geometric mean maintains a consistent weight scale and avoids completely excluding articles with lower reliability scores.

2. Single-stage threshold determination

Regarding the ±0.10 threshold in the single-stage trading framework, this is a fixed threshold applied to the daily aggregate signal. It’s not adjusted during the evaluation period.

3. Novelty and relationship to prior work

We thank the reviewer for highlighting these related studies. While prior work has explored news relevance, novelty filtering, and multidimensional scoring, FinRAIN's contribution is to establish reliability assessment as an independent, reusable component of financial news ingestion, distinct from sentiment prediction and trading decisions. Optional external evidence further supports these assessments.

Specifically, LLT use relevance and novelty thresholds for news selection, while CausalStock cores news primarily for its predictive relevance and market impact. FinRAIN instead evaluates the reliability of the underlying information, recognizing that news may have significant market implications while remaining poorly substantiated. Unlike TrustTrade, which assesses credibility through cross-agent consensus, FinRAIN evaluates individual articles before downstream decision-making.

We will clarify this distinction in the manuscript. Our central contribution is to separate news reliability assessment from downstream prediction, enabling its application across different sentiment models and trading frameworks.

4. Human evaluation

The human evaluation involved two annotators, both authors of the paper, who assessed a random sample of 100 articles. Table 1 reports agreement between FinRAIN scores and human assessments. We did not separately measure inter-annotator agreement between the two human annotators. We recognize that a larger study involving independent annotators would provide further evidence of annotation consistency and construct validity.

Reviewer_z4BzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes FinRAIN, a training-free news ingestion layer for LLM-based trading. An LLM scores each article on five reliability dimensions (market relevance, source attribution, information novelty, headline exaggeration, stakeholder bias). The scoring can optionally be grounded in external sources through pipeline or agentic (tool-based) retrieval. The scores reweight sentiment from five backbones and are inserted into single-stage and two-stage trading frameworks. On a 20-stock, 132-day corpus, the authors report higher sentiment–return Spearman correlation (≈0.06→0.13), higher hit rates, and improved Sharpe ratios. They also include a leave-one-dimension-out ablation and a 100-article human agreement study.

Critical strengths
  1. The problem is real and under-studied. LLM trading pipelines usually treat ingestion as solved, and isolating article reliability as a separate, interpretable interface is a sensible direction. The five-dimension schema is intuitive and close to how analysts actually screen news.
Critical weaknesses
  1. The gains are not attributed to the reliability schema. FinRAIN also changes which articles reach the model: in the case study, the two-stage baseline routes 0 GOOGL articles while FinRAIN routes 25 (Fig. 6). Routing is never ablated, and the “routing only, no rubric” arm in Table 5 has no results in Tables 2–3. There is also no same-LLM baseline with a generic reliability prompt and no plain-RAG baseline. The central claim therefore remains untested.
  2. The evidence is statistically weak and internally inconsistent. No confidence intervals or repeated runs are reported. Over 132 days, the standard error of annualized Sharpe is roughly 1.4 (≈√(252/132) under i.i.d. returns), so most differences in Table 3 are within noise. The agentic configuration is the best on ρ and hit rate, yet it lands near the bottom in two-stage long-short. The paper also contradicts itself in several places:
  • the case study uses a “veracity” score that is not among the five dimensions;
  • scales and names differ between §3 and App. E;
  • the scoring model differs between App. E and Table 2;
  • sector PnL (Tables 7–9) does not reconcile with Table 3;
  • the function mapping the five scores to sentiment weights is never given. The timing of full-text fetches relative to the trading day is also not stated, so look-ahead leakage cannot be ruled out.
Questions for authors
  1. Please report Tables 2–3 for the “routing only, no rubric” arm, a same-LLM baseline with a generic reliability prompt, and a plain-RAG baseline.
  2. How exactly are the five scores converted into weights on backbone sentiment? Please give the formula.
  3. Is the ablation performed by dropping a score column or by re-prompting with four dimensions? How many runs, and with what variance?
  4. What is “veracity” in App. H / Fig. 6, and how does it relate to the five dimensions? Which scorer was used for which tables?
  5. Please add bootstrap CIs for ρ, hit rate, and Sharpe, and reconcile Tables 7–9 with Table 3.
  6. When were full-text pages fetched relative to the trading day, and how do you rule out post-publication updates that mention realized price moves?
  7. How many annotators produced the 100-article labels, and what was inter-annotator agreement?
2 comments

(1/2) We thank the reviewer for the detailed feedback and for recognizing the importance of reliability assessment in financial news ingestion. We appreciate the opportunity to clarify several aspects of our methodology and experimental results.

1. Additional baselines

We thank the reviewer for suggesting these additional controls. To better distinguish the contributions of article routing, generic reliability assessment, and external retrieval, we conducted the three requested baseline experiments.

Table A. Next-session sentiment alignment (Spearman ρ / Directional hit rate %)

MethodFinBERTDistilRoBERTaFinMAClaude Sonnet 4.5GPT-OSS-120b
Baseline.0561 / 53.9.0543 / 52.9.0730 / 53.9.0779 / 53.8.0663 / 53.3
Routing only, no rubric.111 / 54.4.095 / 54.4.116 / 55.0.114 / 55.0.128 / 55.5
Same-LLM, generic reliability prompt.109 / 55.8.101 / 53.6.117 / 54.9.125 / 55.7.119 / 56.0
Plain RAG.122 / 56.3.100 / 54.1.118 / 54.5.124 / 56.7.124 / 57.0
FinRAIN, agentic retrieval.1280 / 55.6.1235 / 55.1.1379 / 55.7.1333 / 57.2.1314 / 56.6

Table B. Two-stage trading performance (Sharpe ratio / PnL in USD)

MethodLong onlyLong shortRound trip
Baseline0.55 / +31.1-0.73 / -54.41.28 / +66.3
Routing only, no rubric1.37 / +75.3-1.23 / -71.01.72 / +78.9
Same-LLM, generic reliability prompt1.42 / +84.30.11 / +1.91.80 / +88.2
Plain RAG1.19 / +69.3-1.74 / -108.41.91 / +94.0
FinRAIN, pipeline retrieval1.60 / +106.8-0.46 / -34.52.50 / +138.2

The additional results show that routing contributes to performance, while FinRAIN achieves the highest sentiment correlation across all five backbones. In downstream trading, FinRAIN outperforms the alternative baselines in long-only and round-trip strategies, though not in long-short. Overall, these findings provide further evidence that structured reliability assessment offers benefits beyond routing and generic reliability prompting.

2. Weighting formulation and ablation

The five reliability scores are normalized to [0, 1] and combined using their geometric mean to obtain an article-level reliability weight: Ri=(∏j=1ksi,j5)1/k,k=5.R_i = \left(\prod_{j=1}^{k}\frac{s_{i,j}}{5}\right)^{1/k}, \qquad k=5. where si,js_{i,j} denotes the reliability score for article ii and dimension jj. This formulation gives each dimension equal importance while allowing lower scores to reduce an article's contribution. The geometric mean maintains a consistent weight scale and avoids completely excluding articles with lower reliability scores.

3. Ablation study

The ablation study is performed by removing one reliability dimension at a time from the article weighting calculation, rather than re-prompting the scoring LLM. Specifically, we retain the original five-dimensional scores and recompute the reliability weight using the geometric mean of the remaining four normalized dimensions. This maintains a comparable weight scale across configurations. The underlying articles, ticker assignments, and sentiment predictions remain unchanged, allowing us to isolate the marginal contribution of each dimension to sentiment aggregation. Each configuration is evaluated once, without independent repeated LLM scoring runs.

4. Explanation of veracity

The "veracity" score shown in Appendix H and Figure 6 originates from an earlier experimental design that used a different reliability scoring schema. It served as a proxy for the credibility of news claims based on source characteristics and uncertainty cues. We subsequently refined this dimension into Source Attribution to provide a clearer, more interpretable assessment of the transparency and credibility of supporting evidence. The outdated terminology was inadvertently retained and will be removed for consistency with the final framework.

(2/2)

5. Statistical uncertainty

We have conducted additional analyses to estimate 95% confidence intervals for directional hit rates and annualized Sharpe ratios. The results show higher point estimates for FinRAIN in the reported comparisons, although the confidence intervals overlap, particularly for Sharpe ratios. We therefore interpret the trading improvements as encouraging empirical evidence rather than statistically conclusive gains. We will incorporate these intervals and clarify the statistical uncertainty in the revised manuscript.

Table C. Directional hit rates (%) with 95% confidence intervals

MethodFinBERTDistilRoBERTaFinMAClaude Sonnet 4.5GPT-OSS-120b
Baseline53.9 [50.9, 57.0]52.9 [49.1, 56.8]53.9 [50.0, 57.8]53.8 [50.2, 57.5]53.3 [49.8, 57.0]
FinRAIN, agentic retrieval55.6 [51.9, 59.2]55.1 [51.4, 58.7]55.7 [52.1, 59.2]57.2 [53.7, 60.6]56.6 [53.3, 59.9]

Table D. Annualized Sharpe ratios with 95% confidence intervals

Framework / MethodLong Only [95% CI]Long Short [95% CI]Round Trip [95% CI]
Single-stage / Baseline-1.07 [-4.62, 2.48]-0.54 [-4.03, 2.99]0.39 [-2.83, 3.65]
Single-stage / FinRAIN, pipeline retrieval0.07 [-3.18, 3.31]0.72 [-3.42, 4.45]0.87 [-1.92, 3.67]
Two-stage / Baseline0.55 [-2.63, 3.96]-0.73 [-3.88, 2.40]1.28 [-2.21, 4.48]
Two-stage / FinRAIN, pipeline retrieval1.60 [-1.27, 4.64]-0.46 [-3.67, 2.78]2.50 [-0.83, 5.61]

6. Temporal validity

News was collected retrospectively, but the sentiment and trading evaluation was constructed to respect the information available at each trading decision. Specifically, for each trading day, only articles published before the market open were eligible for inclusion, while external evidence retrieval was restricted to information available prior to the corresponding article's publication timestamp. These temporal constraints prevent the use of future market information in sentiment estimation and trading decisions.

7. Human annotation

The human evaluation involved two annotators, both authors of the paper, who assessed a random sample of 100 articles. We did not separately measure inter-annotator agreement between the two human annotators. We recognize that a larger study involving independent annotators would provide further evidence of annotation consistency and construct validity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.