acceptodds
Withdrawn from ICLR 2027

Is LLM-Curated Policy Evidence Trustworthy Enough for Causal Analysis? An Audit-then-Propagate Framework Applied to Global Digital-Nomad Visa Data

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) can turn legal and policy documents into structured datasets, but extraction accuracy alone does not show whether the resulting evidence supports a downstream causal conclusion. We introduce an **audit-then-propagate** framework that independently checks policy claims against primary sources and measures how a field-specific error affects staggered-adoption difference-in-differences estimates. In a digital-nomad-visa study spanning 190 jurisdictions, a human audit confirms 33 adopters. Among the LLM pipeline’s 46 determinate program-existence calls, 32 agree with the audit (69.6%; Cohen’s ); 144 jurisdictions remain unresolved by the pipeline’s evidentiary standard. Adoption year matches in five of six fully cross-checked cases, while tax and financial-threshold fields lack independently verified reference values. A 200-iteration semi-synthetic experiment substitutes recorded announcement dates for verified effective dates. This timing error attenuates estimates toward zero by 0.25–0.28 for both two-way fixed effects and a Callaway–Sant’Anna-style estimator, reducing 95% confidence-interval coverage from 72.0% to 50.0% and from 90.0% to 67.5%, respectively. An application using verified dates and national unemployment data produces directionally negative event-time estimates, but none is individually distinguishable from zero; the specification also provides no pre-treatment estimates with which to assess differential trends. The study shows how an auditable curation error can materially change inference, while making the benchmark’s unresolved coverage and the limits of its observational application explicit.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
trustworthy AI, large language models, AI-assisted data curation, causal inference, difference-in-differences, human verification, policy evidence, error propagation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xf51OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes auditing LLM-extracted policy data and then measuring how the remaining errors affect downstream causal estimates. It studies digital-nomad-visa policies across 190 jurisdictions, compares LLM classifications with a separate human audit, and constructs a benchmark containing 33 human-coded adopters. A semi-synthetic experiment substitutes announcement dates for effective-exposure dates and reports attenuation and reduced confidence-interval coverage for TWFE and a CS-style estimator. A real-data application finds negative but statistically inconclusive associations with national unemployment.

Critical strengths
  1. The paper asks a worthwhile question about LLM-based data curation. Tracing an error in a specific field through to an estimate and its uncertainty is more useful than reporting a single extraction-accuracy number, especially when the dataset will be used for policy analysis.

  2. The audit gives a concrete account of several different failure modes. Appendix A.6 distinguishes source-label problems, discovery versus verification, missing dates, and unresolved disagreements. This detail is useful for understanding what a policy-data audit must check, even though the audit itself remains incomplete.

  3. The authors are appropriately cautious about several results. They disclose the 144 unresolved LLM cases, the small 5/6 timing comparison, and the lack of human verification for tax and financial fields. They also avoid interpreting the unemployment estimates as an established causal effect, given the limited cohort support and missing pre-treatment estimates.

Critical weaknesses
  1. The human audit is not a sufficiently reliable reference for the claimed verification. Section 3.3 assigns 131 of 157 human “no” labels because no primary source was found, while the LLM can return Unresolved under similar uncertainty. Thus, the reported agreement compares two different coding rules. Appendix A.6 also lists problems with primary-source labels in the human audit, and Appendix C.2 says the disagreements have not been adjudicated. Please apply consistent uncertainty rules, resolve the material disagreements, and identify which absence-based negatives enter the never-treated control group.

  2. The simulation changes more than the treatment date. Appendix D converts effective dates to the first full exposure year but uses the raw calendar year for announcement dates. Even identical announcement and effective dates after January would therefore produce a one-year shift. How much of the reported +0.25–0.28 attenuation remains when both dates use the same conversion rule? The paper should also list the affected jurisdictions and cohort shifts; currently it cannot establish how many dates were changed.

  3. The CS-style standard-error calculation is not justified. Appendix D takes an observation-weighted RMS of cohort standard errors and divides it by the square root of the number of cohorts. This is not generally the standard error of the weighted aggregate and does not account for covariance from shared controls. Coverage is already only 72% with correct dates, which makes the further coverage loss difficult to interpret. Please rerun this analysis with valid clustered inference and report failed-replicate counts. The explanation in Section 4.3 that undercoverage reflects “wider standard errors” also needs correction.

  4. The reproducibility claims conflict with the rest of the manuscript. The reproducibility statement says the seed, clustering procedure, and noise process are unavailable, while Section 4.3 and Appendix D specify them. The identities of some included and excluded jurisdictions also remain unresolved. Likewise, Tables 1 and 5 describe policy fields as verified despite the stated lack of human verification for tax and financial values. These are consequential inconsistencies: the paper needs one confirmed account of the data, sample, and implementation underlying the reported results.

Questions for authors

See weaknesses 1–4.

Reviewer_htCjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a policy dataset that an LLM extracts from legal documents is reliable enough to feed a causal analysis, and proposes an "audit-then-propagate" framework: audit the extracted fields against primary sources, then push a field-specific error through a staggered difference-in-differences estimator. The case study is digital-nomad visas across 190 jurisdictions. 32 of the 46 determinate LLM calls agree with a single-researcher human audit (69.6%, kappa 0.258), and a 200-iteration semi-synthetic experiment shows that coding treatment at the announcement rather than the effective date attenuates TWFE and Callaway-Sant'Anna-style estimates by 0.25-0.28 and lowers CI coverage.

Critical strengths

The question is the right one and it is rarely asked this directly: extraction accuracy is the wrong summary for data that will feed an identification strategy, and tracing one field-specific error into a specific estimator is the right shape of answer. I also appreciate that the authors do not inflate their coverage: 144 of 190 jurisdictions remain unresolved and they say so plainly, and reporting kappa next to raw agreement is the honest pairing.

Critical weaknesses

The central result does not measure what the paper says it measures. Appendix D states that the verified arm uses a month-aware date-to-year rule while the misdated arm uses the announcement's raw calendar year, and concedes that part of the attenuation may reflect "this date-to-year convention difference rather than the announcement-versus-effective gap alone." Two arms that differ in two ways do not isolate one of them. The fix is mechanical (same conversion in both arms, rerun), but as submitted the headline 0.25-0.28 is not attributable to the error the paper claims to study.

More worrying, the manuscript states in several places that its own analysis artifacts "were not available to us" (the audit file in Section 4.3; the simulation's seed, clustering procedure and noise process in the Reproducibility Statement), yet the main text and Appendix D report a seed of 12345 and the full generating process. One has to wonder who ran the analysis that produced these numbers, and the paper needs to say. The rest compounds this: the 14 LLM-vs-human disagreements have had no adjudication pass, so the "ground truth" is not established; the timing claim rests on 5 of 6 cross-checked cases (Wilson CI 43.6-97.0%); the CS-style estimator already covers at only 72% with correct dates; and the extraction ran through the ChatGPT interface at temperature 1 with no seed, so nobody can rerun it. Finally, I do not see an ICLR contribution here: there is no method or model that generalizes beyond an applied-econometrics audit, and the DiD machinery would get more expert scrutiny at a political methodology venue.

Questions for authors
  1. What exactly does "not available to us" mean, given that the paper reports the seed and the generating process in full?
  2. If both arms use the same month-aware date-to-year conversion, how much of the 0.25-0.28 attenuation survives?
  3. Will the 14 disagreements be adjudicated before any agreement figure is reported?
Reviewer_LcdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into whether LLM extracted policy data is reliable to a good degree for causal analysis - it uses digital nomad visa policies across countries - compares LLM outputs against human - tests how incorrect policy adoption dates can distort the estimation of policy effects on unemployment.

Critical strengths
  • Connects human auditing of LLM-extracted policy data to downstream causal inference, which seams novel,
  • Transparent reporting of unresolved cases, i really liked the transparency. thank you!
Critical weaknesses
  • I find the paper weak on the ground truth: only one human auditor. 131 non-adopters classified through absence of evidence, no???
  • what should be done with unresolved disagreements: 19 human – LLM classification disagreements ???
  • The LLM failed to classify 144 of 190 jurisdictions (76%). doesn't this make the coverage incomplete???
  • I think the paper lacks sufficient validation: only 6 cases timing check. tax, ... income, and fee fields never verified.
  • Why artificial error experiment??
  • Confounded main result: Correct and incorrect dates use different year-conversion rules. The reported bias cannot be attributed solely to LLM error, can it???
Questions for authors

My questions are listed in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.