acceptodds
Under review as a conference paper at ICLR 2027

When Financial Facts Change: Auditing Labels for Point-in-Time Financial Learning

Abstract

Historical financial data are versioned observations rather than immutable rows: later annual reports can change values reported for an earlier period. This creates two hazards for machine learning—invalid revision-derived labels and predictors that silently contain future information. We construct accession-paired observations from the U.S. SEC's public as-filed XBRL data and test whether comparative changes or explicit restatement dimensions recover a regulatory error-correction construct. Across fiscal years 2011–2023, 31,688 of 354,650 matched core facts change, affecting 22.24% of 47,918 company-years. Explicit restatement dimensions cover only 6.17% of changed facts and are absent in the first five years, so their absence is not a stable negative label. We use the SEC's DocumentFinStmtErrorCorrectionFlag as a filing-level reference in two temporal cohorts whose protocols were frozen before label access. Any comparative change has precision 17.26%/12.68% and recall 57.43%/57.78%; the explicit-dimension proxy has precision 66.67%/49.21% and recall 39.60%/34.44% for fiscal years 2023/2024. Both proxies fail their frozen validity thresholds in both years. A retrospective fiscal-year benchmark is not a deployable timing test: its cohort conditions on future availability and its training labels can postdate test prediction times. We reconstruct first-filing cohorts of 3,961 and 4,158 companies without future eligibility filters, preserve unknown labels, and require training-label maturity at each prediction cutoff. The audit distinguishes comparative revision from regulatory error correction and identifies additional cohort and training-label timing requirements for financial ML. Complete score coverage is distinct from complete reference-label coverage. Recovering an earlier training cohort restores score coverage to all 4,158 test companies. On the 3,606 annotated cases, boosting achieves AUROC 0.591, while poor calibration and 552 unknown labels limit the prediction claim.

Primary area:
datasets and benchmarks
Keywords:
financial machine learning, point-in-time data, label validity, SEC XBRL, temporal leakage

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Z3RYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper audits whether changes between accession-paired SEC XBRL observations, or explicit restatement dimensions, recover a filing-level error-correction flag. A 2011–2023 panel quantifies revision frequency and changing availability of dimensional evidence, followed by two sequential proxy-validity audits. The paper then constructs first-filing cohorts, restricts training to labels available at monthly prediction cutoffs, and distinguishes complete score coverage from observed-outcome coverage. An earlier-training extension provides modest ranking evidence, while unknown outcomes and class-weight sensitivity limit the interpretation of predictive performance.

Critical strengths
  1. A concrete longitudinal audit with an explicitly defined reference. Sections 3–4 distinguish accession-level revisions from the filing-level correction flag. The 354,650 matched facts, historical variation in restatement-dimension coverage, and two sequential proxy audits make the label-construction problem measurable. Reporting that the proxies fail the stated screening criteria is useful even without a new predictive model.
  2. Transparent treatment of availability and unfavorable results. Sections 5–6 retain unknown outcomes, enforce monthly training-label maturity, and distinguish scoring all 4,158 companies from evaluating the 3,606 observed outcomes. Section 5.2 also provides a matched March comparison: adding earlier training data improves ranking on that subset while worsening Brier loss. Reporting this tradeoff, the failed overwrite-inflation criterion, and the retrospective nature of the weighting sensitivity strengthens the audit's credibility.
Critical weaknesses
  1. Proxy disagreement has not been sufficiently resolved at the semantic level. The proxies examine ten concepts in one target period, while the reference flag can concern any financial-statement period in the later filing (Sections 3.1–3.2 and 4.2, pp. 3–5). The authors acknowledge this mismatch, but the confusion matrices do not separate non-error revisions, out-of-scope concepts or periods, repeated disclosures, and reference errors. The CSX example validates one accession pair, not the reasons for disagreement across the cohort. The results therefore establish checkbox-recovery failure, while leaving unclear which label-construction lesson follows beyond the chosen proxies' limited coverage.
  2. The practical effects of the different availability safeguards remain insufficiently separated. Appendix A's overwrite contrast operates within a future-selected cohort without enforced training-label maturity; its HGB AP difference is 0.0157 with interval [−0.0160, 0.0621]. The authors correctly do not claim leakage inflation. The revised and historical evaluations also differ in population and training information. Section 5.2's matched March comparison does establish an effect of adding earlier training data, but does not isolate eligibility, label maturity, and feature vintage under the revised protocol. This limits the additional empirical knowledge supplied by the timing audit, rather than demonstrating that the safeguards are unnecessary.
  3. Outcome selection substantially limits the predictive evidence. Of 4,158 test companies, 552 (13.28%) have unknown outcomes, including 538 without an eligible next-Q1 filing (Sections 5–5.1, p. 6). Unknown-label companies have smaller median log assets than observed-label companies, 19.20 versus 20.95. The paper correctly preserves these cases, but performance is still measured on one observed-outcome test cohort with only 90 positives. The prevalence bounds and company bootstrap intervals do not establish robustness of model comparisons to outcome selection. Complete score coverage therefore repairs an operational gap while leaving the population-level predictive interpretation unresolved; the observed size difference alone does not prove a particular missingness mechanism.
Questions for authors
  1. Semantic validation: Can you provide existing source-filing annotations, or a small stratified check of proxy/reference disagreements, identifying the affected concepts and periods? Please distinguish non-error revisions, corrections outside proxy coverage, and repeated disclosures where identifiable. This would determine what the failure rates establish beyond a scope mismatch.
  2. Availability effects: Beyond the matched March earlier-training comparison, which existing records permit varying cohort eligibility, training-label maturity, or feature vintage separately on a stated common population? Please provide a feasible matched contrast with uncertainty, or specify the narrower empirical conclusion if such contrasts are not identifiable. A new model is not needed.
  3. Unknown outcomes: Using the saved scores and observability records, can you show how the principal model comparisons depend on outcome selection under explicit sensitivity assumptions? Please distinguish conclusions supported for the observed-label subset from those, if any, supported for the entire first-filing cohort; the prevalence bound alone cannot answer this question.
  4. Historical reconstruction: Are there checks against original XBRL instances from both early and later periods, beyond the CSX example, that validate the Restatement= extraction and the reported absence of early dimensional evidence? Please identify the archive versions and distinguish reconstruction of as-filed facts from verification of historical feed availability. Existing checks or a small targeted sample would materially affect this concern.
2 comments

Thank you for specifying the checks needed to interpret the audit. We completed source-filing, matched-information, and unknown-outcome analyses using the saved records.

Source-filing evidence. We recovered the SEC index and primary 10-K/10-K/A for all 172 P_dim disagreements, with unique accessions and SHA-256 hashes. Every FP has an explicitly dimensioned changed core fact (20/20 in FY2023 and 32/32 in FY2024). Among FNs, only 18/61 and 21/59 have any changed fact within the ten concepts and target comparative period; the remaining 43/61 and 38/59 contain no changed fact within that narrow coverage. FY2023 disagreements are entirely before the April 11, 2025 C&DI cutoff and FY2024 disagreements entirely after it. A deterministic 20-filing evidence sample, selected by the five lowest predeclared hashes per year/FP-FN stratum, accompanies the complete audit table. These records turn the abstract scope mismatch into a source-document result.

Matched training-information contrast. On the same 1,734 annotated March companies (46 positives), adding the recovered earlier matured training cohort increases HGB AUROC by 0.1589 [0.0527, 0.2674] and AP by 0.0484 [0.0149, 0.1223]; Brier loss increases by 0.0403 [0.0340, 0.0465]. Logistic AUROC increases by 0.0977 [0.0073, 0.1868]. Because the evaluated companies and outcomes are identical, this contrast establishes that the available training information materially changes ranking and calibration.

Unknown-outcome sensitivity. All 4,158 companies receive scores; 3,606 have observed outcomes and 552 remain unknown. On the observed subset, HGB AUROC is 0.591. We assigned the unknown outcomes under explicit scenarios instead of coding them as negatives. With assumed unknown-group prevalence of 5%, HGB AUROC ranges from 0.454 when positives occupy its lowest scores to 0.687 when they occupy its highest scores. The corresponding ranges are 0.377--0.735 at 10% prevalence and 0.299--0.771 at 20%. Random fixed-seed assignments give 0.551, 0.561, and 0.534, respectively. These results quantify how population-level interpretation depends on outcome observability while preserving complete score coverage.

Together, the three checks support the paper's main methodological conclusion: proxy scope, training-information timing, and outcome observability are separate properties and must be audited separately. We will make this structure explicit and add the complete source-file and sensitivity protocols to the revision.

I thank the authors for the responses to my reivews. Collectively, the rebuttal resolved my questions on Q1/Q2. However, the remaining concerns on Q3/Q4 are not adequately resolved from my perspective. I appreciate the transparency and thoroughness of the added checks, but given the remaining methodological and presentation limitations, I will maintain my current rating.

Reviewer_jGFLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper shows that changes in prior financial statement numbers do not predict actual restatements or error corrections. It also reviews constructing leakage-free financial database.

Critical strengths

[1] If the authors release the point-in-time financial statements for all SEC filings, that would be of importance to researchers.

[2] The problem that the authors aim to study is potentially important. Silent financial statement corrections can introduce lookahead bias in predictions.

Critical weaknesses

[1] I am confused by what the authors are trying to establish. The paper is very incoherently written and tries to deliver too many different (but logically unconnected) messages.

I agree that using 'backfilled' numbers in financial tasks can be problematic, but this has already been shown by Lyle et al. (2025). Again, for this one, the authors cite the paper as a contemporaneous 2026 paper, but it was first circulated in 2025.

Also, many studies already try to predict 'restatements' or 'misstatements' from existing financial information. However, I do not see any economic value in trying to predict 'simple error corrections' from current information. Consistent with my conjecture, the authors' classifiers do not deliver usable performance.

Lastly, if the authors want to pursue this line of research, one potential path could be to use 'backfilled' data and 'original' data to show that some prediction results materially change. This would establish the importance of the authors' approach. As written, the authors fail to show that lookahead bias inflates predictive accuracy.

[2] What do the authors wish to show by predicting 'error flags of year t+1' using year t data? Error flags of year t+1 indicate that there was at least one error in previous filings, and this could have been years before t. In this case, using year t data to predict the year t+1 flag makes no sense. The authors acknowledge this on page 6, but this seriously undermines the paper's intended contribution. The authors should confine the flags to year t errors only.

[3] The models that the authors develop have poor performance. It is unclear what the readers can do with the models. It would be better if the authors could clearly explain how these negative results can serve the academic community.

They might also consider trying more advanced prediction methods.

[4] Also, it is very problematic to compare the pre-selected 10 financial statement items while the SEC flags are at the filing level. With this scope mismatch, it is very expected that the models will not perform well.

[5] The paper's writing is very difficult to follow. The chapters are not logically consistent. The tests serve different purposes. The authors should substantially improve the writing of the overall manuscript.

Questions for authors

Please respond to the Weakness section above.

2 comments

Thank you for pressing us to state the paper's purpose more clearly. We will reorganize it around one claim: point-in-time financial learning requires validation of both the label construct and the time at which every training label becomes observable. The proxy audit and the prediction audit test these two requirements in sequence, with the prediction model serving as a controlled illustration of the timing protocol.

The next-filing flag is used as a filing-level disclosure available at a dated future accession, not as evidence that an accounting error occurred specifically in year t. The prediction exercise asks whether a score for that later disclosure can be produced for every first-filing company using only information and matured labels available at each cutoff. This is why unknown outcomes remain unknown and why training-label maturity is enforced month by month.

Two completed controls make the practical lesson concrete. First, on the same 1,529 FY2023 companies, the t0 filing flag ranks the later flag better than HGB: AUROC 0.655 versus 0.621 and AP 0.123 versus 0.0617. Second, on an identical set of 1,734 annotated March companies, adding the recovered earlier matured training cohort changes HGB AUROC by +0.159 [0.053, 0.267] and AP by +0.0484 [0.0149, 0.1223], while Brier loss changes by +0.0403 [0.0340, 0.0465]. Thus the information available for training materially changes both ranking and calibration on a fixed evaluation population.

The source-filing audit also makes the role of the ten-concept proxy explicit. Across all 172 P_dim disagreements, every FP contains an explicitly dimensioned changed fact within the proxy scope, whereas 43/61 FY2023 FNs and 38/59 FY2024 FNs contain no changed fact among those concepts in the target comparative period. The comparison therefore demonstrates a construct-and-scope mismatch, rather than measuring the accuracy of an error detector.

We will correct the first-circulation year of Lyle et al. to 2025 and rewrite the manuscript so that the longitudinal label audit, timing protocol, and limited predictive illustration form a single progression. The resulting resource is useful precisely because it prevents a stronger classifier from being evaluated on a temporally or semantically invalid target.

Reviewer_jGFLOpenReview ↗

Acknowledgement

I appreciate the authors' response. However, I remain unconvinced that the paper clears the bar at the conference.

The response leaves my central concerns unresolved.

[1] “Timing protocol,” “label maturity,” and a “controlled illustration” do little to sharpen what readers actually learn from this paper. Redefining the outcome as a filing-level disclosure does little to enhance its economic value.

[2] The authors fail to deliver requested additional analyses.

[3] Describing the comparison as a “construct-and-scope mismatch” simply re-iterates the problem I identified without providing any resolution.

[4] Finally, promises to rewrite and reorganize the manuscript do not resolve these conceptual and empirical flaws.

Overall, the response is unnecessarily opaque, and I still cannot identify a clear research question with coherent tests to support it. I retain my score.

Reviewer_M9sROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two hazards in point-in-time financial learning: treating revised XBRL values as error-correction labels, and training on values that later filings overwrite. It combines a 2011–2023 historical panel with FY2023/FY2024 proxy audits using filings through Q1 2026, and evaluates logistic regression, histogram gradient boosting, and training-prevalence predictions under time-aligned cohorts and mature-label cutoffs.

Critical strengths
  1. Auditing two easily adopted revision proxies against the SEC's error-correction flag, in two cohorts with thresholds fixed in advance, gives concrete numbers on a common labeling shortcut. The paper rightly treats the flag as a reference construct rather than ground truth.
  2. Failed gates and adverse results are reported openly, and the released predictions reproduce Tables 2–3 exactly, which makes independent re-analysis possible.
Critical weaknesses
  1. The audit has not shown how its findings change conclusions in an established financial-ML dataset or task. A controlled reconstruction of one such task would substantiate its broader contribution.
  2. Proxy audit stops at confusion matrices. No P_dim disagreement (20/32 FP, 61/59 FN) is checked against filings, and the reference is unexamined: the 10-K box is conditional on Section 12(b) registration, and the April 2025 SEC C&DIs (Q104.20, Q104.22) fall between the two cohorts' flag filings. It would help to adjudicate disagreements and stratify by 12(b) status and pre/post-April 2025.
  3. A simple persistence baseline is missing. Using the t0 filing's own flag as the predictor already matches HGB's AUROC (≈0.591 vs. 0.591); reporting it would show how much signal the model adds beyond this one bit, which matters more here than comparing with newer models.
Questions for authors

See weaknesses.

2 comments

Thank you for identifying the persistence baseline and source-filing checks as the most informative additions. We completed both analyses.

On the common FY2023 subset for which the t0 flag, matured next-filing label, and chronologically fitted models are all available (1,529 companies; 53 positives), the t0 filing flag attains AP 0.123 [0.066, 0.203], AUROC 0.655 [0.591, 0.724], and Brier 0.0510 [0.0405, 0.0621]. HGB attains AP 0.0617 [0.0402, 0.1130], AUROC 0.621 [0.540, 0.703], and Brier 0.0428 [0.0348, 0.0514]. In a paired 2,000-resample company bootstrap, HGB minus persistence is -0.0614 [-0.1393, 0.0230] in AP, -0.0347 [-0.1353, 0.0688] in AUROC, and -0.00822 [-0.01896, 0.00194] in Brier. Persistence is the stronger ranking baseline in this cohort, while HGB has the lower Brier score. We will report persistence as the primary predictive comparator and use the prediction section to demonstrate the timing protocol.

We also recovered the SEC filing index and primary 10-K/10-K/A document for all 172 P_dim disagreements: 20/61 FY2023 FP/FN and 32/59 FY2024 FP/FN. Primary documents were selected from the SEC index by filing type, and all 172 accessions and SHA-256 hashes are unique. Every FP contains an explicitly dimensioned changed core fact (20/20 and 32/32). By contrast, 43/61 FY2023 FNs and 38/59 FY2024 FNs contain no changed fact among the ten pre-specified concepts in the target comparative period. Thus, 81 of 120 FNs contain no changed fact within the proxy's stated fact-period scope. The FY2023 filings all precede the April 11, 2025 C&DI cutoff and the FY2024 filings all follow it, so the two temporal strata are now reported explicitly. The SecurityExchangeName tag is present in 16/20 and 26/32 FPs and 61/61 and 58/59 FNs; we use it as an auditable filing field rather than a legal determination of Section 12(b) status.

These additions sharpen the paper's central result: comparative revisions and explicit dimensions are narrow, time-varying observables, while the SEC checkbox is a filing-level disclosure. Their disagreement must be analyzed at the source-filing level before either proxy is used as a machine-learning label.

Thank you for the response and the additional analyses. However, (W1) the impact on conclusions in an established financial-ML task still remains unclear, and (W2) the reasons for disagreement remain insufficiently resolved. I also share Reviewer jGFL's concern about the construct-and-scope mismatch. I therefore maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.