- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper audits whether changes between accession-paired SEC XBRL observations, or explicit restatement dimensions, recover a filing-level error-correction flag. A 2011–2023 panel quantifies revision frequency and changing availability of dimensional evidence, followed by two sequential proxy-validity audits. The paper then constructs first-filing cohorts, restricts training to labels available at monthly prediction cutoffs, and distinguishes complete score coverage from observed-outcome coverage. An earlier-training extension provides modest ranking evidence, while unknown outcomes and class-weight sensitivity limit the interpretation of predictive performance.
Critical strengths
- A concrete longitudinal audit with an explicitly defined reference. Sections 3–4 distinguish accession-level revisions from the filing-level correction flag. The 354,650 matched facts, historical variation in restatement-dimension coverage, and two sequential proxy audits make the label-construction problem measurable. Reporting that the proxies fail the stated screening criteria is useful even without a new predictive model.
- Transparent treatment of availability and unfavorable results. Sections 5–6 retain unknown outcomes, enforce monthly training-label maturity, and distinguish scoring all 4,158 companies from evaluating the 3,606 observed outcomes. Section 5.2 also provides a matched March comparison: adding earlier training data improves ranking on that subset while worsening Brier loss. Reporting this tradeoff, the failed overwrite-inflation criterion, and the retrospective nature of the weighting sensitivity strengthens the audit's credibility.
Critical weaknesses
- Proxy disagreement has not been sufficiently resolved at the semantic level. The proxies examine ten concepts in one target period, while the reference flag can concern any financial-statement period in the later filing (Sections 3.1–3.2 and 4.2, pp. 3–5). The authors acknowledge this mismatch, but the confusion matrices do not separate non-error revisions, out-of-scope concepts or periods, repeated disclosures, and reference errors. The CSX example validates one accession pair, not the reasons for disagreement across the cohort. The results therefore establish checkbox-recovery failure, while leaving unclear which label-construction lesson follows beyond the chosen proxies' limited coverage.
- The practical effects of the different availability safeguards remain insufficiently separated. Appendix A's overwrite contrast operates within a future-selected cohort without enforced training-label maturity; its HGB AP difference is 0.0157 with interval [−0.0160, 0.0621]. The authors correctly do not claim leakage inflation. The revised and historical evaluations also differ in population and training information. Section 5.2's matched March comparison does establish an effect of adding earlier training data, but does not isolate eligibility, label maturity, and feature vintage under the revised protocol. This limits the additional empirical knowledge supplied by the timing audit, rather than demonstrating that the safeguards are unnecessary.
- Outcome selection substantially limits the predictive evidence. Of 4,158 test companies, 552 (13.28%) have unknown outcomes, including 538 without an eligible next-Q1 filing (Sections 5–5.1, p. 6). Unknown-label companies have smaller median log assets than observed-label companies, 19.20 versus 20.95. The paper correctly preserves these cases, but performance is still measured on one observed-outcome test cohort with only 90 positives. The prevalence bounds and company bootstrap intervals do not establish robustness of model comparisons to outcome selection. Complete score coverage therefore repairs an operational gap while leaving the population-level predictive interpretation unresolved; the observed size difference alone does not prove a particular missingness mechanism.
Questions for authors
- Semantic validation: Can you provide existing source-filing annotations, or a small stratified check of proxy/reference disagreements, identifying the affected concepts and periods? Please distinguish non-error revisions, corrections outside proxy coverage, and repeated disclosures where identifiable. This would determine what the failure rates establish beyond a scope mismatch.
- Availability effects: Beyond the matched March earlier-training comparison, which existing records permit varying cohort eligibility, training-label maturity, or feature vintage separately on a stated common population? Please provide a feasible matched contrast with uncertainty, or specify the narrower empirical conclusion if such contrasts are not identifiable. A new model is not needed.
- Unknown outcomes: Using the saved scores and observability records, can you show how the principal model comparisons depend on outcome selection under explicit sensitivity assumptions? Please distinguish conclusions supported for the observed-label subset from those, if any, supported for the entire first-filing cohort; the prevalence bound alone cannot answer this question.
- Historical reconstruction: Are there checks against original XBRL instances from both early and later periods, beyond the CSX example, that validate the Restatement= extraction and the reported absence of early dimensional evidence? Please identify the archive versions and distinguish reconstruction of as-filed facts from verification of historical feed availability. Existing checks or a small targeted sample would materially affect this concern.
2 comments
I thank the authors for the responses to my reivews. Collectively, the rebuttal resolved my questions on Q1/Q2. However, the remaining concerns on Q3/Q4 are not adequately resolved from my perspective. I appreciate the transparency and thoroughness of the added checks, but given the remaining methodological and presentation limitations, I will maintain my current rating.
Thank you for specifying the checks needed to interpret the audit. We completed source-filing, matched-information, and unknown-outcome analyses using the saved records.
Source-filing evidence. We recovered the SEC index and primary 10-K/10-K/A for all 172 P_dim disagreements, with unique accessions and SHA-256 hashes. Every FP has an explicitly dimensioned changed core fact (20/20 in FY2023 and 32/32 in FY2024). Among FNs, only 18/61 and 21/59 have any changed fact within the ten concepts and target comparative period; the remaining 43/61 and 38/59 contain no changed fact within that narrow coverage. FY2023 disagreements are entirely before the April 11, 2025 C&DI cutoff and FY2024 disagreements entirely after it. A deterministic 20-filing evidence sample, selected by the five lowest predeclared hashes per year/FP-FN stratum, accompanies the complete audit table. These records turn the abstract scope mismatch into a source-document result.
Matched training-information contrast. On the same 1,734 annotated March companies (46 positives), adding the recovered earlier matured training cohort increases HGB AUROC by 0.1589 [0.0527, 0.2674] and AP by 0.0484 [0.0149, 0.1223]; Brier loss increases by 0.0403 [0.0340, 0.0465]. Logistic AUROC increases by 0.0977 [0.0073, 0.1868]. Because the evaluated companies and outcomes are identical, this contrast establishes that the available training information materially changes ranking and calibration.
Unknown-outcome sensitivity. All 4,158 companies receive scores; 3,606 have observed outcomes and 552 remain unknown. On the observed subset, HGB AUROC is 0.591. We assigned the unknown outcomes under explicit scenarios instead of coding them as negatives. With assumed unknown-group prevalence of 5%, HGB AUROC ranges from 0.454 when positives occupy its lowest scores to 0.687 when they occupy its highest scores. The corresponding ranges are 0.377--0.735 at 10% prevalence and 0.299--0.771 at 20%. Random fixed-seed assignments give 0.551, 0.561, and 0.534, respectively. These results quantify how population-level interpretation depends on outcome observability while preserving complete score coverage.
Together, the three checks support the paper's main methodological conclusion: proxy scope, training-information timing, and outcome observability are separate properties and must be audited separately. We will make this structure explicit and add the complete source-file and sensitivity protocols to the revision.