- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents a resource for studying reported AI-related harms. Its contributions are: (1) a taxonomy of AI harms; (2) a dataset of 3k+ harm cases extracted from news reports (2021–2026), each linked to its original source; (3) a knowledge graph of these events, together with a learned temporal hypergraph model for distinguishing observed AI harm events from corrupted ones; and (4) a dynamic interactive, country-level map of AI harms. The data pipeline is anchored by 56-case manually verified reference set, and the hypergraph model is evaluated against seven knowledge-graph and hypergraph baselines on a chronological held-out set of 174 events, using AUC (plus AUPRC, F1 and MCC; Table 1a), with an ablation study (Table 1b).
Critical strengths
- Scope and organization of the resource. The paper organizes reported AI harms into a structured taxonomy and dataset whose coverage goes beyond prior resources such as CSET's AI Harm Framework or the MIT AI Risk Repository in the range of harm types it covers. The value of the paper is that a structured, up-to-date record of AI harms could be useful for both researchers and policymakers.
- Traceability. Each event is linked to its original news source, so individual entries can be checked against the original source by humans. This matters for my assessment because it makes the data-quality concerns below easier to address in principle, because errors are auditable.
Critical weaknesses
-
Insufficient assessment of data quality. The curation pipeline is well documented, but the quality of the resulting data is not adequately quantified. (a) For the 1,162 human-annotated reports, the paper does not state who the annotators were or report inter-annotator agreement. (b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall. (c) The paper describes the 56-case reference set as “a rigorous 56-case ground truth”, but I could not find any quantification supporting this. The text (“a human reviewer manually collected and fully verified 56 cases”) suggests a single annotator, so we don’t know whether these cases would be subject to disagreement between annotators. Since the data is one of the paper’s main contributions, this is my central concern: without these measurements, I cannot judge how far to trust the case counts and the distributional findings built on them (Sections 4.1 and 4.3).
-
Evaluation of the temporal hypergraph model. (a) The appendix (App. A, “Fair comparison”) states that the baseline’s (HYPER’s) published checkpoint is fine-tuned “on the same test set”. Taken literally, this means the baseline was trained on test data, which makes the comparison uninterpretable. Such leakage should favor the baseline, so if it still underperforms, this needs explaining. Could the authors clarify whether this is a typo and describe the exact protocol? (b) In the ablation study (Table 1b), removing any single component causes a large performance drop (from 92.03 AUC for the full model to between 44.41 and 62.00, with one variant below chance). Such uniformly large drops are unusual and make me want to rule out an issue in the experimental setup. Could the authors double-check these results and explain why each component is this critical?
Questions for authors
Beyond the questions raised under Critical Weaknesses: (Clarification) The paper describes “a pipeline that converts news reports into structured, temporal, and geolocated AI harm events.” How does this pipeline differ methodologically from standard event extraction and automatic knowledge graph construction approaches? The answer affects how much of the pipeline I count as a technical contribution rather than an application of existing techniques.
3 comments
Response
I thank the authors for their answer. In short, I feel that my central concern about data quality was not addressed, and will therefore keep my score. Bellow this paragraph, you can find my reasoning for this decision.
Leakage: Clarified and resolved.
Annotators: I believe that this concern was partially addressed. While the clarification of three annotators is indeed reassuring, the authors still didn't say what was the level of disagreement, which is very important for a dataset contribution.
Precision/recall of the classifier: I wrote in my review "(b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall." This concern was not addressed.
56-case ground truth: I wrote in my review "(c) The paper describes the 56-case reference set as “a rigorous 56-case ground truth”, but I could not find any quantification supporting this." This concern was also not addressed.
Ablations: I think that the authors' explanation of the drop in numbers in the ablation study does not explain the phenomena well. The authors did not report double-checking this or reruning anything. One example of a sanity check would be, instead of relying on a single random seed, a simple check of several seeds (which would rule out noise). Of course, even this would not catch a systematic bug in the code, as such systematic problem would propagate throughout the different random seeds, which should also be checked subsequently.
We thank the reviewer for the follow-up and for pressing on these points, which are essential for a dataset contribution.
"I believe that this concern was partially addressed. While the clarification of three annotators is indeed reassuring, the authors still didn't say what was the level of disagreement, which is very important for a dataset contribution."
Thank you for your reply. We want to clarify that we did not encounter any discrepancies during the annotation verification process.
The reference set of 1,162 reports was constructed through extraction and a full-verification workflow rather than parallel independent annotation. Specifically, two annotators independently processed approximately half of the records. They directly mapped explicit fields from the LLM-classifier JSON records (including report ID, harm category, location, event date, and original evidence span) and extracted the structured event roles (event type, AI system, organization, affected group, action, and consequence) strictly based on the original report text. Any information not explicitly stated in the source text was left blank. Subsequently, a third annotator conducted a 100% verification audit of all 1,162 records against their corresponding source JSON inputs. Since the annotation process relied on extracting explicitly available information from the JSON records, the third annotator's full audit confirmed that there were zero discrepancies between the extracted data and the JSON files. Because of that we do not have any disagreement.
We have described the annotation and verification procedure in Appendix D.1 (Human-annotated reference set) and summarized it in Section 3.2 (Validation of automated curation) of the revised manuscript. Both places now also explain why inter-annotator agreement (IAA) scores are not reported.
"I wrote in my review '(b) The LLM-based classification step is not evaluated in terms of accuracy, precision, or recall.' This concern was not addressed."
We agree this has not yet been evaluated against human labels. As an interim check, a second LLM (GPT) independently re-classified all 99 +972 Magazine articles under the same protocol. It agreed with our classifier on keep-or-discard for 83 of 99 articles (84%, Cohen's κ = 0.67), and 26 of the 33 articles both models admitted (79%) shared at least one subcategory. This shows consistency between models, not accuracy.
We acknowledge that classifier precision and recall have not been measured against human labels, and we will state this explicitly as a limitation in the revision. We will also state that the seven largest GDELT sources were classified mainly from title and subtitle.
"I wrote in my review '(c) The paper describes the 56-case reference set as "a rigorous 56-case ground truth", but I could not find any quantification supporting this.' This concern was also not addressed."
We agree, and we thank the reviewer for pressing this point. The 56 cases were collected and verified by one reviewer at the civil-society organization where the prototype was developed. Because only one person annotated them, no agreement measure exists, and the phrase "a rigorous 56-case ground truth" (Sec. 3.2) overstates what we have. We will replace it with "a 56-case reference set verified by one reviewer from the civil-society organization".
We also clarify their role. The 56 cases were used to define and test the taxonomy and inclusion criteria by hand, before the automated pipeline was built. They were not used to evaluate the LLM classifier or the pipeline. The cases are separate from the 2,686 automated cases, and some lie outside our corpus entirely (e.g., COMPAS, 2016).
"I think that the authors' explanation of the drop in numbers in the ablation study does not explain the phenomena well. The authors did not report double-checking this or rerunning anything. One example of a sanity check would be, instead of relying on a single random seed, a simple check of several seeds (which would rule out noise)....."
We have not rerun the ablations with multiple seeds within the discussion period, and we do not claim to have verified them. Accordingly, the revision no longer states effect sizes. Section 4.2 now says only that, in single-seed component ablations, removing any mechanism lowers AUC; the ablations indicate component sensitivity, not effect sizes. We have removed the sentence that interpreted the joint contribution of the components. The full model's stability is supported separately by the chronological cross-validation in Appendix D.3, whose folds are whole calendar years (fold AUCs 0.902, 0.978, 0.974), but that does not address the ablation variants.
We thank the reviewer for the careful and detailed review. Any changes or additions in response to your review are highlighted in blue in the revised paper.
We thank the reviewer for the careful reading and constructive feedback. We address each point below.
Thank you for raising this point. The 1,162 reports were annotated by three annotators from our team on a shared annotation sheet, with each annotator extracting the structured event fields (event type, the seven roles, date and evidence) for their assigned reports. Each annotator’s extractions were then reviewed against the source report by another annotator. These annotations serve as the reference labels for evaluating automatic extraction and as the gold event graph for PHTKG (Appendix D.1). Further controls apply. Every case links to its source article (Section 3.4), so each label is auditable. The classifier applies a written inclusion rule with an occurrence-status gate (Appendix C.6).
We thank the reviewer for flagging this sentence in Appendix A, which was ambiguous. HYPER was not trained on test data. We started from the official pretrained HYPER checkpoint and fine-tuned it on the training split only, matching the budget given to the other baselines. HYPER was then evaluated on the same chronological test split and the same corrupted negatives as PHTKG and the other baselines, so all methods use identical training data and identical held-out test events. The test split was used only for evaluation. We will revise Appendix A to state the checkpoint, the fine-tuning budget and the evaluation protocol exactly.
The reviewer is right that the Table 1b drops are large, and the caption states that each variant is a single run at seed 42. The design of the benchmark makes the sensitivity expected. Corruptions include a directional organization↔affected-group swap (25%) and same-role substitution (55%) (Appendix D.3). A variant without explicit role parameters cannot separate swapped events, and Appendix F shows that one surface value under two roles has cosine similarity −0.121. Appendix F also shows that changing only the event year changes the representation (cosine −0.129 between 2018 and 2024). All variants use the same split, negatives, and configuration as the full model. Removing state carryover also removes the entity history on which the other components depend. We will state in Section 4.2 that the ablations are single-seed experiments and indicate component sensitivity rather than effect sizes..
We thank the reviewer again for the helpful feedback.
We thank the reviewer for this question. Keyword filtering, LLM-based extraction and NLI verification are established techniques, and we do not claim them as contributions. The pipeline departs from standard event extraction and automatic knowledge graph construction in three respects.
Unit of extraction. Standard event extraction (e.g., ACE, GDELT/CAMEO) assumes a dated triggering event. Our schema takes harm as the unit and assigns each case an occurrence status and an acute or chronic type. This admits chronic harms that lack a single trigger; they make up approximately one third of the dataset.
Event representation. Automatic KG construction methods such as AutoSchemaKG and SHARP decompose events into binary triples. We represent each event as a single seven-role hyperedge whose entity states are carried forward across years and trained with a recurrence objective. Decomposing the same events into triples reduces AUC from 92.0 to 58.8 (Table 1b).
Geolocation. Rather than geocoding every event, our placement procedure abstains when no territorial locus exists. The 193 global or platform-based cases are retained in the data but not mapped, and every case remains traceable to its evidence span and source.
The contribution therefore lies in the schema, the representation and the placement procedure rather than in the extraction techniques. We will state this distinction explicitly in Section 3.
The clarifications provided on the annotation process, experimental protocol, and methodological contributions aim to address your concerns and strengthen the paper. We believe these revisions improve the transparency of the evaluation and better highlight the paper's contributions, and we would greatly appreciate your reconsideration of the assessment.