acceptodds
Under review as a conference paper at ICLR 2027

AI Harms: Taxonomy, Dataset and Real-World Map

Abstract

Artificial intelligence (AI) harms are increasingly reported worldwide, yet no global infrastructure systematically maps their distribution. In contrast, humanitarian crises are monitored through established networks that continuously collect, structure, and visualize evidence on global maps. Evidence of AI harm, however, remains scattered across news reports. Existing AI-incident resources are also largely manually curated, focus on incidents instead of harms, and list cases without automatically locating or mapping them. As a result, chronic and infrastructural AI harms are largely missing, and the global distribution of AI harm remains invisible. To address this, we present the first automated AI Harm Map, a traceable pipeline that turns news reports into a public global map. It rests on a novel harm-centered taxonomy, anchored to 56 human-verified cases and 1,162 human-annotated harm reports. Building on this reference dataset, we scale coverage through automated retrieval, keyword-gated filtering, and LLM-based classification over a 2021–2026 news corpus, yielding 3,038 AI harm cases. To preserve the relational, temporal, and data provenance structure of these cases, we introduce PHTKG, a data provenance-aware temporal hypergraph that jointly models multi-role relationships, temporal dynamics, and evidence reliability, outperforming graph and hypergraph baselines on a chronological benchmark. On a chronological held-out benchmark of 174 events, our PHTKG outperforms the strongest of seven baselines by 4.68 AUC points. The learned representations generate the public interactive AI Harm Map, geolocating 2,845 of 3,038 cases across 81 countries; 193 cases cannot be assigned to a single country, including regional and global cases. Our map reveals that subcategory prevalence varies sharply by geography: weapons-related harms are least US-centered (35% US vs. 49%–65% in other regions) and clustered in conflict-affected regions (Iran, Palestine, Russia, and Ukraine), a pattern that holds when using only locations named in the reports. The dynamic AI Harm Map is available at https://anonymous.4open.science/w/AI_Harm_map-3991/Module_02/visualization/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Harm taxonomy, Harm Map, AI Governance, Knowledge Graphs, PHTKG, temporal hypergraph

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qd1qOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a resource for studying reported AI-related harms. Its contributions are: (1) a taxonomy of AI harms; (2) a dataset of 3k+ harm cases extracted from news reports (2021–2026), each linked to its original source; (3) a knowledge graph of these events, together with a learned temporal hypergraph model for distinguishing observed AI harm events from corrupted ones; and (4) a dynamic interactive, country-level map of AI harms. The data pipeline is anchored by 56-case manually verified reference set, and the hypergraph model is evaluated against seven knowledge-graph and hypergraph baselines on a chronological held-out set of 174 events, using AUC (plus AUPRC, F1 and MCC; Table 1a), with an ablation study (Table 1b).

Critical strengths
  • Scope and organization of the resource. The paper organizes reported AI harms into a structured taxonomy and dataset whose coverage goes beyond prior resources such as CSET's AI Harm Framework or the MIT AI Risk Repository in the range of harm types it covers. The value of the paper is that a structured, up-to-date record of AI harms could be useful for both researchers and policymakers.
  • Traceability. Each event is linked to its original news source, so individual entries can be checked against the original source by humans. This matters for my assessment because it makes the data-quality concerns below easier to address in principle, because errors are auditable.
Critical weaknesses
  • Insufficient assessment of data quality. The curation pipeline is well documented, but the quality of the resulting data is not adequately quantified. (a) For the 1,162 human-annotated reports, the paper does not state who the annotators were or report inter-annotator agreement. (b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall. (c) The paper describes the 56-case reference set as “a rigorous 56-case ground truth”, but I could not find any quantification supporting this. The text (“a human reviewer manually collected and fully verified 56 cases”) suggests a single annotator, so we don’t know whether these cases would be subject to disagreement between annotators. Since the data is one of the paper’s main contributions, this is my central concern: without these measurements, I cannot judge how far to trust the case counts and the distributional findings built on them (Sections 4.1 and 4.3).

  • Evaluation of the temporal hypergraph model. (a) The appendix (App. A, “Fair comparison”) states that the baseline’s (HYPER’s) published checkpoint is fine-tuned “on the same test set”. Taken literally, this means the baseline was trained on test data, which makes the comparison uninterpretable. Such leakage should favor the baseline, so if it still underperforms, this needs explaining. Could the authors clarify whether this is a typo and describe the exact protocol? (b) In the ablation study (Table 1b), removing any single component causes a large performance drop (from 92.03 AUC for the full model to between 44.41 and 62.00, with one variant below chance). Such uniformly large drops are unusual and make me want to rule out an issue in the experimental setup. Could the authors double-check these results and explain why each component is this critical?

Questions for authors

Beyond the questions raised under Critical Weaknesses: (Clarification) The paper describes “a pipeline that converts news reports into structured, temporal, and geolocated AI harm events.” How does this pipeline differ methodologically from standard event extraction and automatic knowledge graph construction approaches? The answer affects how much of the pipeline I count as a technical contribution rather than an application of existing techniques.

3 comments

We thank the reviewer for the careful reading and constructive feedback. We address each point below.

“(a) For the 1,162 human-annotated reports, the paper does not state who the annotators were or report inter-annotator agreement. (b) ... (c) ...”

Thank you for raising this point. The 1,162 reports were annotated by three annotators from our team on a shared annotation sheet, with each annotator extracting the structured event fields (event type, the seven roles, date and evidence) for their assigned reports. Each annotator’s extractions were then reviewed against the source report by another annotator. These annotations serve as the reference labels for evaluating automatic extraction and as the gold event graph for PHTKG (Appendix D.1). Further controls apply. Every case links to its source article (Section 3.4), so each label is auditable. The classifier applies a written inclusion rule with an occurrence-status gate (Appendix C.6).

“Could the authors clarify whether this is a typo and describe the exact protocol?” ... “...removing any single component causes a large performance drop ...”

We thank the reviewer for flagging this sentence in Appendix A, which was ambiguous. HYPER was not trained on test data. We started from the official pretrained HYPER checkpoint and fine-tuned it on the training split only, matching the budget given to the other baselines. HYPER was then evaluated on the same chronological test split and the same corrupted negatives as PHTKG and the other baselines, so all methods use identical training data and identical held-out test events. The test split was used only for evaluation. We will revise Appendix A to state the checkpoint, the fine-tuning budget and the evaluation protocol exactly.

The reviewer is right that the Table 1b drops are large, and the caption states that each variant is a single run at seed 42. The design of the benchmark makes the sensitivity expected. Corruptions include a directional organization↔affected-group swap (25%) and same-role substitution (55%) (Appendix D.3). A variant without explicit role parameters cannot separate swapped events, and Appendix F shows that one surface value under two roles has cosine similarity −0.121. Appendix F also shows that changing only the event year changes the representation (cosine −0.129 between 2018 and 2024). All variants use the same split, negatives, and configuration as the full model. Removing state carryover also removes the entity history on which the other components depend. We will state in Section 4.2 that the ablations are single-seed experiments and indicate component sensitivity rather than effect sizes..

We thank the reviewer again for the helpful feedback.

“How does this pipeline differ methodologically from standard event extraction and automatic knowledge graph construction approaches?”

We thank the reviewer for this question. Keyword filtering, LLM-based extraction and NLI verification are established techniques, and we do not claim them as contributions. The pipeline departs from standard event extraction and automatic knowledge graph construction in three respects.

Unit of extraction. Standard event extraction (e.g., ACE, GDELT/CAMEO) assumes a dated triggering event. Our schema takes harm as the unit and assigns each case an occurrence status and an acute or chronic type. This admits chronic harms that lack a single trigger; they make up approximately one third of the dataset.

Event representation. Automatic KG construction methods such as AutoSchemaKG and SHARP decompose events into binary triples. We represent each event as a single seven-role hyperedge whose entity states are carried forward across years and trained with a recurrence objective. Decomposing the same events into triples reduces AUC from 92.0 to 58.8 (Table 1b).

Geolocation. Rather than geocoding every event, our placement procedure abstains when no territorial locus exists. The 193 global or platform-based cases are retained in the data but not mapped, and every case remains traceable to its evidence span and source.

The contribution therefore lies in the schema, the representation and the placement procedure rather than in the extraction techniques. We will state this distinction explicitly in Section 3.

The clarifications provided on the annotation process, experimental protocol, and methodological contributions aim to address your concerns and strengthen the paper. We believe these revisions improve the transparency of the evaluation and better highlight the paper's contributions, and we would greatly appreciate your reconsideration of the assessment.

Reviewer_qd1qOpenReview ↗

Response

I thank the authors for their answer. In short, I feel that my central concern about data quality was not addressed, and will therefore keep my score. Bellow this paragraph, you can find my reasoning for this decision.

Leakage: Clarified and resolved.

Annotators: I believe that this concern was partially addressed. While the clarification of three annotators is indeed reassuring, the authors still didn't say what was the level of disagreement, which is very important for a dataset contribution.

Precision/recall of the classifier: I wrote in my review "(b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall." This concern was not addressed.

56-case ground truth: I wrote in my review "(c) The paper describes the 56-case reference set as “a rigorous 56-case ground truth”, but I could not find any quantification supporting this." This concern was also not addressed.

Ablations: I think that the authors' explanation of the drop in numbers in the ablation study does not explain the phenomena well. The authors did not report double-checking this or reruning anything. One example of a sanity check would be, instead of relying on a single random seed, a simple check of several seeds (which would rule out noise). Of course, even this would not catch a systematic bug in the code, as such systematic problem would propagate throughout the different random seeds, which should also be checked subsequently.

We thank the reviewer for the follow-up and for pressing on these points, which are essential for a dataset contribution.

"I believe that this concern was partially addressed. While the clarification of three annotators is indeed reassuring, the authors still didn't say what was the level of disagreement, which is very important for a dataset contribution."

Thank you for your reply. We want to clarify that we did not encounter any discrepancies during the annotation verification process.

The reference set of 1,162 reports was constructed through extraction and a full-verification workflow rather than parallel independent annotation. Specifically, two annotators independently processed approximately half of the records. They directly mapped explicit fields from the LLM-classifier JSON records (including report ID, harm category, location, event date, and original evidence span) and extracted the structured event roles (event type, AI system, organization, affected group, action, and consequence) strictly based on the original report text. Any information not explicitly stated in the source text was left blank. Subsequently, a third annotator conducted a 100% verification audit of all 1,162 records against their corresponding source JSON inputs. Since the annotation process relied on extracting explicitly available information from the JSON records, the third annotator's full audit confirmed that there were zero discrepancies between the extracted data and the JSON files. Because of that we do not have any disagreement.

We have described the annotation and verification procedure in Appendix D.1 (Human-annotated reference set) and summarized it in Section 3.2 (Validation of automated curation) of the revised manuscript. Both places now also explain why inter-annotator agreement (IAA) scores are not reported.

"I wrote in my review '(b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall.' This concern was not addressed."

We agree this has not yet been evaluated against human labels. As an interim check, a second LLM (GPT) independently re-classified all 99 +972 Magazine articles under the same protocol. It agreed with our classifier on keep-or-discard for 83 of 99 articles (84%, Cohen's κ = 0.67), and 26 of the 33 articles both models admitted (79%) shared at least one subcategory. This shows consistency between models, not accuracy.

We acknowledge that classifier precision and recall have not been measured against human labels, and we will state this explicitly as a limitation in the revision. We will also state that the seven largest GDELT sources were classified mainly from title and subtitle.

"I wrote in my review '(c) The paper describes the 56-case reference set as "a rigorous 56-case ground truth", but I could not find any quantification supporting this.' This concern was also not addressed."

We agree, and we thank the reviewer for pressing this point. The 56 cases were collected and verified by one reviewer at the civil-society organization where the prototype was developed. Because only one person annotated them, no agreement measure exists, and the phrase "a rigorous 56-case ground truth" (Sec. 3.2) overstates what we have. We will replace it with "a 56-case reference set verified by one reviewer from the civil-society organization".

We also clarify their role. The 56 cases were used to define and test the taxonomy and inclusion criteria by hand, before the automated pipeline was built. They were not used to evaluate the LLM classifier or the pipeline. The cases are separate from the 2,686 automated cases, and some lie outside our corpus entirely (e.g., COMPAS, 2016).

"I think that the authors' explanation of the drop in numbers in the ablation study does not explain the phenomena well. The authors did not report double-checking this or rerunning anything. One example of a sanity check would be, instead of relying on a single random seed, a simple check of several seeds (which would rule out noise)....."

We have not rerun the ablations with multiple seeds within the discussion period, and we do not claim to have verified them. Accordingly, the revision no longer states effect sizes. Section 4.2 now says only that, in single-seed component ablations, removing any mechanism lowers AUC; the ablations indicate component sensitivity, not effect sizes. We have removed the sentence that interpreted the joint contribution of the components. The full model's stability is supported separately by the chronological cross-validation in Appendix D.3, whose folds are whole calendar years (fold AUCs 0.902, 0.978, 0.974), but that does not address the ablation variants.

We thank the reviewer for the careful and detailed review. Any changes or additions in response to your review are highlighted in blue in the revised paper.

Reviewer_TccAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper collects reports of AI harms from news articles and proposes a taxonomy with five main categories and 22 subcategories. Building on 56 manually verified cases and 1,162 manually annotated reports, the authors process news articles from 2021–2026. The system first filters the articles and then uses a frozen Qwen2.5-3B-Instruct model to extract one event from each article. The authors propose PHTKG, a temporal hypergraph model that learns relationships among events and their evolution over time, to retrieve similar events and predict whether an entity will reappear in the following year.

The authors construct separate graphs from 1,162 manually annotated events and 1,159 automatically extracted events to train PHTKG. Observed events serve as positive examples. Negative examples are generated by replacing a value with another value from the same role, swapping the organization and affected-group roles, or borrowing a role value from another event. Given the complete event fields, year, and confidence score, the model outputs a compatibility score and is trained to score positive examples higher than negative ones. An auxiliary task predicts whether an entity will reappear in the corpus in the following year, with labels derived directly from the yearly records.

Events in each graph are sorted by year and record ID. The main evaluation measures the model's ability to distinguish held-out events from artificial negative examples using AUC and compares it with HYPER and other baselines. The paper also reports next-year entity recurrence prediction, ablation studies, and examples of similar-event retrieval. On 174 held-out automatically extracted events, PHTKG achieves an AUC of 92.03 ± 3.30, exceeding HYPER by 4.68 percentage points.

Critical strengths

Collecting and organizing records of AI harms has practical value. The authors release the data, code, and map implementation.

Critical weaknesses

First, the training tasks are disconnected from the research goal. PHTKG receives complete event fields and is trained to distinguish observed events from artificial negative examples and predict whether entities will reappear in the following year. The paper does not explain how this model helps users analyze AI harms. A high AUC on artificial negative examples does not demonstrate more accurate data or more useful relationships among events.

Second, the released dataset contains many duplicate records. The file module_03/predicted_v2.json contains 3,038 records but only 2,686 distinct source URLs. There are 352 pairs of records sharing a URL, all from The Guardian. Each pair has the same article text, evidence spans, date, and harm category. In 290 pairs, the event fields are identical. The other 62 pairs contain differences, including paraphrases and conflicting extractions. For example, R2150 and R2502 place the same location, West Footscray, in Australia and the United States, respectively. The authors need to explain these duplicates and differences and recompute the statistics after deduplication.

Third, the main experiment does not use a strictly chronological split. The main notebook sorts events by year and record ID, without sorting by date within each year. For example, R046 in the validation set is dated July 31, 2026, while R520 in the test set is dated January 1, 2026. The training set also contains events that occur later than some validation events. This split cannot establish generalization to future events.

Fourth, the choice of extraction model is not justified, and data quality is not adequately validated. The authors use Qwen2.5-3B-Instruct, although Qwen2.5 was released in September 2024. The paper does not explain why it uses this model rather than a newer one, or why it chooses the 3B version rather than a larger variant or a stronger commercial model. It also provides no comparison of extraction performance. The extracted fields directly determine the inputs to the graph model, so extraction errors affect model training and map-based analysis. The authors should compare extraction accuracy using newer and larger models and examine whether PHTKG's results hold when the extraction model changes.

Fifth, the model size and choice are not justified. The released PHTKG model for the automatically extracted graph has approximately 910,000 parameters, far fewer than BERT-base's approximately 110 million. The paper does not explain why it chooses a model of this size rather than further training a pretrained BERT model. The current task is to distinguish observed events from artificial negative examples. The same event fields could be fed into BERT to fine-tune a classifier and evaluated using the same data splits and negative examples. The authors need to demonstrate what advantages hypergraph and temporal modeling offer over BERT.

Questions for authors

How does the proposed model validate the quality of the constructed dataset? The current evaluation only tests whether it can distinguish dataset entries from programmatically generated corrupted examples. How do the authors validate the conversion from source text into structured event records?

2 comments

We thank the reviewer for the careful reading and constructive feedback. We address each point below.

“The paper does not explain how this model helps users analyze AI harms. A high AUC on artificial negative examples does not demonstrate more accurate data or more useful relationships among events.”

Thank you for this comment. We agree that AUC on corrupted events measures event compatibility and not data accuracy, and Section 4.2 scopes the claim to “distinguishing observed AI harm events from corrupted counterparts”. PHTKG’s outputs are part of the released resource: in the released predicted_v2.json, every one of the 3,038 records that feed the map carries a 128-dimensional event embedding, a temporal representation, five nearest events and next-year recurrence probabilities for the AI system and harm category (Figure 1, Appendix D.1, record R001). Users obtain from these (1) related cases through similar-event retrieval (Table A11) and (2) recurrence signals per entity (Table A12, AUC 0.666–0.945 across all four transitions). Compatibility training shapes the embedding, and the recurrence head supplies the second output. We will add to Section 4.2 and the Table 1 caption that Table 1 evaluates event compatibility, not extraction accuracy, which is evaluated in Appendix D.1, and add to Section 4.4 that retrieval is illustrated qualitatively in Table A11 and not benchmarked.

“The released dataset contains many duplicate records. The file module_03/predicted_v2.json contains 3,038 records but only 2,686 distinct source URLs.”

Thank you for checking the released file so carefully. You are right, and this was an error on our side. The batch of 352 admitted Guardian articles was merged into the dataset twice. The second copy received new IDs (R2195–R2546, duplicating R1843–R2194), and the extractor processed each copy separately. The dataset therefore has 2,686 unique cases, not 3,038. The classification itself was not affected: the paper’s count of 2,689 admitted articles is correct, but the case count built on it was inflated.

Because the extractor ran on each copy separately, the 62 differing pairs are two independent extractions of the same article, not two different events. Your West Footscray example is one of them. We have removed the duplicates and kept the first copy of each pair (R1843–R2194). We checked the retained record of each of the 62 differing pairs against its article; in your example, the retained record (R2150) correctly places West Footscray in Australia.

We used the 352 pairs as a consistency test of the extractor. The two runs agree on location in 94.3% of pairs and on 88.9–100% of fields; all event fields are identical in 82.4% of pairs. We will report this in the paper.

“...the main experiment does not use a strictly chronological split ... This split cannot establish generalization to future events.”

Thank you for this careful check. The split is chronological at the temporal resolution of the model. Each event carries its year τ, PHTKG processes events in yearly steps (Section 3.3), and the Limitations section states the yearly resolution. Ordering by report ID within a year therefore supplies no temporal information that the model uses. The reviewer’s R046/R520 example is correct as an ordering of dates inside one year, but the model sees both as the same year. The strict test is the chronological cross-validation in Appendix D.3. Its folds are whole calendar years, with code-level assertions that max(train year) < min(validation year) < min(test year). On predicted events, fold AUCs are 0.902, 0.978 and 0.974 (n_test = 195, 338, 304). Proposed revision (wording only): we will add to the Table 1 caption and Section 4.2 “(yearly resolution; events within a year ordered by report ID)”, and point to the fold results in Appendix D.3 from Section 4.2.

“The paper does not explain why it uses this model rather than a newer one, or why it chooses the 3B version rather than a larger variant or a stronger commercial model.”

Thank you for this comment. We agree that we do not compare extractors, and we will state this in the Limitations. We used Qwen2.5-3B-Instruct because its open weights allow local deployment and reproducible runs within our computational resources, which limited us to a model of this size. Extraction is verified in three stages (Appendix D.1) and evaluated field by field against the 1,162 human annotations (Section 3.3). PHTKG is also trained and evaluated on gold-annotated events, which are independent of the extractor, and on predicted events, with high chronological-CV AUC on both (Appendix D.3). We will add to the Limitations that, under our resource constraints, PHTKG’s results are conditional on one frozen extractor, and that evaluating larger or newer extractors is future work.

“The authors need to demonstrate what advantages hypergraph and temporal modeling offer over BERT.”

Thank you for this suggestion. We agree that a text-encoder baseline would be informative and none is included; we will say so in Section 4.4. We agree that a text-encoder baseline would provide an informative additional comparison, and we acknowledge that such a baseline is not included in the current experiments. However, BERT is not a directly equivalent comparison for our prediction task. PHTKG performs link prediction over a role-indexed entity vocabulary: one organization, AI system or affected group is one node across events, entity states carry forward by year, and a recurrence head predicts whether a given entity reappears the next year (Section 3.3, Table A12). A classifier over one event’s text has no entity identity or history. Table 1a compares against the models built for this setting, hypergraph and set-based methods over the same seven role fields, and Table 1b ablates joint structure, time, provenance and recurrence. The model is small because its parameters are per-role vocabulary embeddings (d = 64, 2 layers, Appendix D.3), and parameter count does not measure suitability.

“How do the authors validate the conversion from source text into structured event records?”

Thank you for this question. PHTKG is not proposed as a validator of the dataset, and Section 3.3 uses the gold and predicted graphs only to separate representation quality from extraction noise. Table 1 evaluates event compatibility, and the conversion from source text to structured event records is validated separately. Each extracted event passes schema, NLI and harm-category verification (Appendix D.1), and the extractor is compared field by field against the 1,162 human annotations using exact match and token-level F1 (Section 3.3).

We hope our clarifications have addressed your concerns and strengthened the paper. We introduced a novel AI harm taxonomy, a new dataset of real-world AI harms, PHTKG model, and, to our knowledge, the first automated, dynamic AI Harm Map. We believe these contributions offer a useful foundation for documenting and tracking AI harms over time, supporting transparency and evidence-based action. We would be grateful if you would reconsider your score in light of these revisions. Thank you again for your time and constructive feedback.

Reviewer_tvUpOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper aims to better understanding of chronic and infrastructural AI harms by converting news reports into a global map, based on a taxonomy built from human-annotated cases and harm reports.

Contributions:

  • AI harm taxonomy
  • Dataset of AI harm cases
  • PHTKG: hypergraph model looking at role, temporal dynamics, and evidence reliability
  • AI harm map

Validation:

  • Human-annotated cases and harm reports
  • Held-out benchmark of 174 events to validate PHTKG
  • Qualitative analysis of AI harm map
Critical strengths
  • Important topic and interesting approach: absolutely worth tracking harms of AI (this is the primary thing pushing me towards accept in the face of the weaknesses below, which I am very concerned about)

  • Nice to see PHTKG alongside some qualitative analysis of the findings

Critical weaknesses
  • To build an AI harm taxonomy and map that can be automatically updated, I think it's important to validate that automatic curation process. This currently feels undersupported (see questions).

  • Limited discussion on how this would continue to work if deployed - e.g., how would it be shared with the organizations it is meant to support, how those organizations could use this in the context of reported limitations.

(nit pick - titles in the AI harms map as you hover around are difficult to parse)

post rebuttal: I feel that given other reviewers' concerns and the response, which suggested there was more work to be done (e.g., two-annotator evaluation of admitted and rejected articles which I believe was completed in lines 250-260, but it's not clear to me why there is a range 88.9–100% of fields, how these compare to the automated checks, and how repeatable this workflow is given the remaining disagreements in the 352 duplicate articles; there is also promised to be extensions to the pipeline to non-English sources, social media, and multimodal evidence to expand geographic coverage), I think this is a great topic but recommend resubmitting in the future.

Questions for authors
  • How does your harm taxonomy compare to existing harm taxonomies, e.g., that in the AHA paper? https://arxiv.org/abs/2306.03280 (see A.3, page 21 in particular).

  • AI Incident Database (AIID) and other existing resources are criticized for geographic bias, but the map also shows a strong grouping in Europe and the United States. How does your approach handle geographic bias better than AIID and others, and why do you still see geographic bias? Are there geographic biases in the way you curate a corpus of news articles to process?

  • In considering harms vs. incidents, why not include social media data here, and/or why it's not possible to incorporate incidents into your work? Section 3.1 or related work would benefit from a clear definition of incidents/how they differ from harms.

  • Why is geolocation prioritized more in the interface vs., for example, a histogram of harm types in your taxonomy (like Figure 4)? Especially given the global harms that are sometimes found, and especially to try to find chronic issues as highlighted in the aims of the work.

  • How did you validate the 3k cases? Did humans select 56 retrieved articles for manual review, or completely manually select them? I know some of this might be in the appendix, but slightly more detail in the main body would be useful to consider how you validate your claims.

  • Why select 20% of retrieved articles? If running this automatically, would you similarly only process 20% of returned documents? Line 231 ish

  • Much of the motivation comes from civil society organizations and/or humanitarian work. Were there any efforts to discuss what these organizations are looking for?

4 comments

We thank the reviewer for the careful reading and constructive feedback. We address each point below.

“To build an AI harm taxonomy and map that can be automatically updated, I think it’s important to validate that automatic curation process. This currently feels undersupported (see questions).”

Thank you; we agree that an automatically updated map needs its curation step validated directly.

Some checks were already in place. Every record keeps its matched keywords, evidence span and source link, so each case can be traced and checked by hand. A record is admitted only after it passes format, source-consistency (NLI) and taxonomy checks. A second model independently re-did the classification for one full source (+972 Magazine, 99 articles), and an audit of the country field corrected 17 of 755 records. Articles that fit no subcategory were counted rather than forced into one, and that is how we found the gap that became subcategory 5.4.

We also found that 352 articles had been extracted twice by mistake (see our reply to Reviewer TccA). The two runs agreed on location in 94.3% of cases and on 88.9–100% of the other fields.

For the revision, we will add a two-annotator evaluation of admitted and rejected articles (precision, recall, agreement), and field-level accuracy against the 1,162 annotated reports. Each future release will also include a published human audit of a random sample of new cases.

“Limited discussion on how this would continue to work if deployed - e.g., how would it be shared with the organizations it is meant to support...”

We thank the reviewer for this helpful comment. We agree that the paper should better explain how organizations can access and use the map beyond its release. We will add a Using the map paragraph to Section 3.4 and update the interface to display the source coverage and last-updated date. Our aim is to maintain the map as a sustained public resource, similar to humanitarian monitoring tools such as the WFP HungerMap and FEWS NET. Regular updates will require hosting, verification, and maintenance support, which the authors are actively seeking. The released dataset and code will be released independently of that support.

Proposed paragraph for Section 3.4:

“The map serves the public interest by bringing together fragmented evidence of AI harm into a shared, source-linked record, enabling researchers, practitioners, and policymakers to examine patterns across cases, regions, and time. Each case links to its labels, source outlet, and original report for verification, while the 193 cases without a single geographic location are listed separately and retained in the released dataset. The map supports case discovery, cross-case analysis, and efforts to document recurring harms and inform governance and accountability, including by helping lawyers and investigators find where primary evidence may be located. However, as it captures reported cases rather than actual harm incidence, it cannot estimate prevalence or establish causation or liability. Users should therefore compare shares within years or categories, interpret empty regions as gaps in reporting, and treat small counts as lower bounds. The map will display its source coverage (27 English-language domains) and the US share of geographically placed cases (57%).”

“...titles in the AI harms map as you hover around are difficult to parse”

We thank the reviewer for this observation. The hover titles are currently generated automatically from article URLs, which makes them hard to read. We will fix this in the revised map. Each title will show the article’s original headline, followed by a one-line summary of the case: AI system, harm subcategory and affected group.

“How does your harm taxonomy compare to existing harm taxonomies, e.g., that in the AHA paper?”

Thank you for the pointer. We will include AHA as a reference framework in Related Work. AHA supports anticipation by generating hypothetical harms through vignettes to help practitioners assess planned deployments before implementation. Our framework supports documentation by structuring reported harms that pass an occurrence-status gate. The two are complementary: documented cases reveal which anticipated harms materialize, while anticipatory tools can guide future monitoring. AHA organizes harms by their effects on stakeholders, whereas our taxonomy follows the AI supply chain and deployment context, recording affected interests separately. For example, our subcategories 1.1, 1.2, and 1.6 overlap with AHA’s representational, allocational, and quality-of-service categories. However, AHA focuses on problematic model behavior and excludes intended or malicious use. It does not cover our infrastructure (2.1–2.3), labor supply-chain (3.1), or military-deployment (5.1–5.4) categories. Additionally, our acute/chronic tag is a subcategory-level attribute, whereas AHA’s one-time/accumulated dimension is a prompt manipulation.

“How does your approach handle geographic bias better than AIID and others, and why do you still see geographic bias? Are there geographic biases in the way you curate a corpus of news articles to process?”

Thank you for raising this important question. Existing resources often depend on human identification, submission, or editorial curation of cases. In contrast, our automated extraction module discovers candidate AI harm cases directly from open web sources, removing manual case submission as a selection bias step. However, it does not eliminate geographic bias, as the pipeline still depends on publicly reported, accessible information. Harms in regions with limited press coverage or documentation capacity may therefore remain underrepresented. Our corpus also draws on selected, text-based news domains, potentially underrepresenting non-English sources and regions outside their coverage. In response, we will revise the Introduction and Literature Review, replacing the statements on manual collection and English-language coverage with the following:

(i) “This process produces verified entries but bounds how many cases can be recorded and makes coverage dependent on the incidents identified and submitted by individuals.” (ii) “These approaches underrepresent chronic and infrastructural harms, while manual collection limits scalability and makes case coverage dependent on the individuals who identify and submit incidents, potentially introducing source and geographic biases.”

We also plan to extend the pipeline to non-English sources, social media, and multimodal evidence to improve geographical coverage and capture community-reported harms.

“In considering harms vs. incidents, why not include social media data here ...? Section 3.1 or related work would benefit from a clear definition of incidents/how they differ from harms.”

Thank you for these valuable suggestions.

Regarding social media: Our current pipeline processes text-based news articles only. We prioritized these sources because editorially reviewed reports provide a more readily verifiable basis for establishing harm occurrence and corroborating cases. Although social media can capture community-reported harms, it requires additional verification of provenance, authenticity, and potential manipulation. We have therefore included social media integration and multimodal analysis in Future Work, with source-specific verification procedures and the same occurrence-status and evidence requirements.

Regarding incidents and harms: We will add explicit definitions to the paper. An incident refers to a discrete event with an identifiable triggering act at a specific point in time, whereas a harm refers to a documented adverse effect on a protected interest, such as life, livelihood, the environment, or dignity, linked to an AI system.

Unlike incident-based datasets, our taxonomy uses AI harm rather than the incident as its unit of analysis, allowing it to capture both acute and chronic harms. Incidents remain valuable sources for identifying and corroborating harm cases. However, our infrastructural perspective also captures cumulative harms that may not stem from a single identifiable event, such as long-term water depletion from data-center operations or wage suppression through algorithmic scheduling. Each case is assigned an occurrence status, and we include only harms that have occurred or are supported by sufficient evidence to be considered probable, distinguishing them from speculative risks.

“Why is geolocation prioritized more in the interface vs., for example, a histogram of harm types in your taxonomy (like Figure 4)?”

We thank the reviewer. We prioritize geography because the map aims to show where AI harms are documented and where reporting gaps remain. Unlike a histogram, it reveals differences in the geographic distribution of harm categories and highlights underrepresented regions. The 193 cases without a single geographic location are listed separately rather than assigned an arbitrary country. The histogram of harm types is also retained (Figure 4), alongside category and subcategory filters. Regarding chronic harm, in the revision we will add a category-by-year view beside the map, with a note that shares reflect reporting, to show how documented harms evolve over time. Since event type is fixed per subcategory, a separate acute/chronic histogram would largely repeat the existing taxonomy.

“How did you validate the 3k cases? Did humans select 56 retrieved articles for manual review, or completely manually select them?”

Thank you for this question. The 56 reference cases were not drawn from the retrieved articles. A human reviewer collected and verified them manually before, and independently of, the automated pipeline. Some lie outside our corpus entirely; for example, the COMPAS case (ProPublica, 2016) predates our 2021–2026 window and comes from an outlet we do not crawl. We used these cases to define and test the inclusion criteria and the taxonomy, not to measure the accuracy of the roughly 3k automated cases. The revision states this explicitly.

The automated cases are validated separately, as described in our reply to W1. Each record passes format, source-consistency and taxonomy checks. A second model audited the classification for one full source and the country field. Two independent extraction runs agree on 88.9–100% of fields. The revision adds a two-annotator evaluation of admitted and rejected articles, and field-level accuracy against 1,162 human-annotated reports.

We agree this belongs in the main text, and we will add a short summary of the manual protocol and the validation results into Section 3.

“Why select 20% of retrieved articles? If running this automatically, would you similarly only process 20% of returned documents?”

Thank you for catching this; our wording was misleading. The 20% is not a sample we chose. It is the share of articles that passed our inclusion rule. Every article that passes the keyword filter goes to the classifier, and the classifier admits only those that document an AI system causing harm. Of the 11,628 WordPress and GDELT articles, 2,337 (20%) met that rule. The rest were rejected for reasons such as AI being mentioned only in passing, a keyword used in another sense, or a harm in which AI played no causal role.

The rate is an outcome, not a setting, and it varies a lot by source: 5% for biometricupdate.com, a trade publication where almost every article mentions biometrics, 7% for The Guardian and 61% for The Intercept (Table A7). In an automated deployment, every retrieved article would be processed in the same way, and the admission rate would vary with the sources.

In the revision we will replace “we select 2,337 (20%)” with “2,337 (20%) met the inclusion criteria”.

“Much of the motivation comes from civil society organizations and/or humanitarian work. Were there any efforts to discuss what these organizations are looking for?”

Thanks for raising this question. The prototype was developed within a civil society organization, shared with human rights and policy practitioners (including UN agencies and international digital rights organizations), and refined through their feedback. Across more than 50 conferences and workshops, practitioners have repeatedly highlighted the lack of shared infrastructure for documenting and addressing AI harms collaboratively. Their feedback informed key design decisions: practitioners called for treating data centers as infrastructure, supporting the Energy / Land category, while those working on AI and the military highlighted that related harms are largely absent from major AI forums, supporting the Deployment Context / Weapons category. An international organization also raised gender disparities in AI harms, which our affected-group field (Appendix G.9) can support, although a gender-disaggregated view is not yet built and remains future work.

This engagement consisted of prototype feedback and conversations at practitioner events, not a structured needs assessment. We plan a structured consultation with civil society and grassroots organizations, linked to the community-sourcing workstream, to establish their needs before accepting community-submitted accounts. We cannot disclose the organizations’ names during double-blind review.

We hope that our clarifications, additional validation plans, and proposed improvements to the map address your concerns. We believe these changes strengthen the framework's reliability and practical value for documenting AI harms, and we would be grateful if you would consider revisiting your assessment in light of them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Key Contributions

We have carefully addressed all reviewer comments and strengthened the manuscript accordingly. We believe this work makes a unique and timely contribution to the global community, where a definition and taxonomy of AI harms are missing, as well as an automated, solid workflow for identifying AI harms in real time. To address this gap, we introduce the first automated pipeline to track AI harms that have already occurred. Our work makes four key contributions: (1) a novel taxonomy of 5 categories and 22 subcategories, shaped by discussions across 50+ conferences and workshops; (2) a new dataset of 2,686 real-world AI harm cases from a 2021–2026 news corpus, anchored to 56 human-verified seed cases; (3) PHTKG, a data-provenance-aware temporal hypergraph model; and (4) a public, dynamic, automated AI Harm Map geolocating 2,527 cases across 81 countries, with traceable source links.

We invite the AC to explore our first live AI Harm World Map https://anonymous.4open.science/w/AI_Harm_map-3991/Module_02/visualization/, which makes this evidence publicly accessible and supports broader efforts toward global AI accountability and human rights.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.