- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes auditing LLM-extracted policy data and then measuring how the remaining errors affect downstream causal estimates. It studies digital-nomad-visa policies across 190 jurisdictions, compares LLM classifications with a separate human audit, and constructs a benchmark containing 33 human-coded adopters. A semi-synthetic experiment substitutes announcement dates for effective-exposure dates and reports attenuation and reduced confidence-interval coverage for TWFE and a CS-style estimator. A real-data application finds negative but statistically inconclusive associations with national unemployment.
Critical strengths
-
The paper asks a worthwhile question about LLM-based data curation. Tracing an error in a specific field through to an estimate and its uncertainty is more useful than reporting a single extraction-accuracy number, especially when the dataset will be used for policy analysis.
-
The audit gives a concrete account of several different failure modes. Appendix A.6 distinguishes source-label problems, discovery versus verification, missing dates, and unresolved disagreements. This detail is useful for understanding what a policy-data audit must check, even though the audit itself remains incomplete.
-
The authors are appropriately cautious about several results. They disclose the 144 unresolved LLM cases, the small 5/6 timing comparison, and the lack of human verification for tax and financial fields. They also avoid interpreting the unemployment estimates as an established causal effect, given the limited cohort support and missing pre-treatment estimates.
Critical weaknesses
-
The human audit is not a sufficiently reliable reference for the claimed verification. Section 3.3 assigns 131 of 157 human “no” labels because no primary source was found, while the LLM can return Unresolved under similar uncertainty. Thus, the reported agreement compares two different coding rules. Appendix A.6 also lists problems with primary-source labels in the human audit, and Appendix C.2 says the disagreements have not been adjudicated. Please apply consistent uncertainty rules, resolve the material disagreements, and identify which absence-based negatives enter the never-treated control group.
-
The simulation changes more than the treatment date. Appendix D converts effective dates to the first full exposure year but uses the raw calendar year for announcement dates. Even identical announcement and effective dates after January would therefore produce a one-year shift. How much of the reported +0.25–0.28 attenuation remains when both dates use the same conversion rule? The paper should also list the affected jurisdictions and cohort shifts; currently it cannot establish how many dates were changed.
-
The CS-style standard-error calculation is not justified. Appendix D takes an observation-weighted RMS of cohort standard errors and divides it by the square root of the number of cohorts. This is not generally the standard error of the weighted aggregate and does not account for covariance from shared controls. Coverage is already only 72% with correct dates, which makes the further coverage loss difficult to interpret. Please rerun this analysis with valid clustered inference and report failed-replicate counts. The explanation in Section 4.3 that undercoverage reflects “wider standard errors” also needs correction.
-
The reproducibility claims conflict with the rest of the manuscript. The reproducibility statement says the seed, clustering procedure, and noise process are unavailable, while Section 4.3 and Appendix D specify them. The identities of some included and excluded jurisdictions also remain unresolved. Likewise, Tables 1 and 5 describe policy fields as verified despite the stated lack of human verification for tax and financial values. These are consequential inconsistencies: the paper needs one confirmed account of the data, sample, and implementation underlying the reported results.
Questions for authors
See weaknesses 1–4.