acceptodds
Under review as a conference paper at ICLR 2027

Do third-party frontier AI evaluations matter? Analyzing publicly available evidence of company responses to evaluations

Abstract

Government AI institutes and independent evaluators study frontier AI systems to identify, evaluate, and better understand potential risks. However, identifying and evaluating a risk itself does not mitigate it. Here, we examine publicly documented company and policy responses to third-party evaluation findings of frontier AI systems. We identified 1,146 distinct findings about frontier models across 457 AI evaluation reports published by government institutes, academia, and evaluation organizations between March 2023 and August 2026. Of these, we classified 233 as adverse empirical findings concerning identifiable models or developers for which a company response could be reasonably expected. Among 190 findings that met predefined significant-risk criteria, 152 findings (80%) lacked a specific and proportionate publicly documented company response for risk mitigation. Collectively, the results suggest that, based on available information, there is limited public evidence that third-party evaluations of frontier AI systems consistently lead to proportionate, attributable, and verifiable responses. Comparing the accountability structures currently in place of third-party evaluation organizations with those of established oversight bodies in other industries, including food, drugs, energy, transportation, and finance, we conclude that formal mechanisms for access, response, remediation, verification, and follow-up inspired by governance in these other sectors could increase the consistency with which AI evaluations lead to more proportionate action.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Frontier AI, third-party evaluation, model auditing, AI accountability, Technical AI Governance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BGkBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper explores whether third party model evaluations are actually acted on or lead to changes by model deployers. They find that the majority of findings are not acted upon (80%).

Critical strengths

This is a really well-written and insightful paper. The layout and flow of the narrative has clearly had a lot of time devoted to it and is a very pleasant read.

With the common push that we need more independent third party evaluations, It's often not questioned how effective they are -- and the this paper tackles that head on, which is a novel area of investigation.

The methods are well-documented, honest, and robust. The sample of organisations and evaluations considered is broad, categorisation of responses to model findings is clear. The analyis here is simple, but performed to a high standard.

The findings themselves are important, and potentially quite impactful. I think the unfortunate starkness of the results is something that needs to be raised to the AI community and have a light shined on it. Resolving this lack of action is important for AI to be ensured safe.

Critical weaknesses

I think ultimately the only reason that I would recommend against acceptance is that this is not necessarily in scope for a main track ICLR submission. I do believe it is of interest to many members of the ICLR community, but this paper is more of a commentary / position about the current state of affairs in AI rather than itself advancing AI as a scientific discipline.

Questions for authors

How should the AI community go about addressing the issues that you highlight in your paper? This is briefly touched upon in section 5, but more concrete or opinionated suggestions would be helpful.

1 comment

Thank you for taking the time to review our paper and share your feedback. We address your comments and questions below.

Question1: Recommendations for the AI community

How should the AI community go about addressing the issues that you highlight in your paper? This is briefly touched upon in section 5, but more concrete or opinionated suggestions would be helpful.

We are not offering a formal position or making recommendations in this paper. However, if actors working on AI governance wanted to increase the likelihood of proportionate responses to discovered harms, our results might support an independent body to track significant-risk findings, set deadlines for written responses from the responsible developers, and record whether reported fixes have been independently verified. Such a body could report cases to a government authority when a developer crosses the set response deadline or a significant risk remains even after follow-up. If a law were to establish this process, it should specify which developers owe a duty to respond, which regulator can enforce that duty, and what penalties apply if there is noncompliance by the developer.

Three existing laws provide precedents for different parts of this mechanism:

  • Response deadline: 49 U.S.C. § 1135 requires the U.S. Secretary of Transportation to respond in writing within 90 days to NTSB recommendations directed to the Secretary.
  • Escalation to a named authority: Article 90 of the EU AI Act allows its scientific panel to alert the AI Office when it has reason to suspect that a general-purpose AI model poses a concrete risk at Union level.
  • Enforcement: California’s SB 53 provides a way for reporting critical safety incidents to Cal OES and authorizes the Attorney General to enforce civil penalties for noncompliance.

An independent body or an established, enforced process involving these suggestions could be used for addressing the issues associated with significant risks and could make follow-up more consistent.

Action 1.1: We have added these suggestions in Appendix H.

Weakness 1: Eligibility of paper for ICLR

I think ultimately the only reason that I would recommend against acceptance is that this is not necessarily in scope for a main track ICLR submission. I do believe it is of interest to many members of the ICLR community, but this paper is more of a commentary / position about the current state of affairs in AI rather than itself advancing AI as a scientific discipline.

We believe it reasonably fits with ICLR. The ICLR 2027 Call for Papers lists “societal considerations including fairness, safety, privacy” and “datasets and benchmarks” among its subject areas. The Reviewer Guide mentions whether a paper contributes “new knowledge and sufficient value to the community” and explicitly recognizes empirical results as a basis for that contribution.

Our main contribution is an empirical study of what happens after third-party evaluations have identified risks in frontier AI systems. In our study, we found concrete evidence of no public response to any of the significant-risk findings by most frontier developers, concerning their models. Our study also provides a baseline for assessing whether future evaluation practices lead to more consistent public follow-up.

Question 2.1: If you remain concerned that this paper is marginal for ICLR's focus, could we propose (1) revision of score as if you had no concern, and (2) separately you can comment your concern in a comment to the AC whose formal responsibility it is to assess fit?

Reviewer_CGPKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines publicly documented company and policy responses to third-party frontier AI evaluations. It compiles 1,146 findings from 457 reports published between March 2023 and August 2026, identifying 233 trackable findings, including 190 classified as significant-risk. Using AI-assisted searches and human review, the authors code risk severity, company responses, and policy uptake. They report that 80% of significant-risk findings lack a public company response meeting their predefined proportionality criteria. Supporting analyses include source-linked evidence, alternative counting units, and adjustments for clustering within reports. The paper also compares AI evaluation institutions with seven oversight bodies in other sectors to motivate formal response and follow-up mechanisms. Its conclusions concern public documentation rather than unobserved private action or causal effects on risk.

Critical strengths
  1. The paper systematically links evaluation findings to documented company responses and policy uptake, moving beyond aggregate descriptions of evaluation activity. The source-linked corpus provides a concrete basis for studying what follows risk identification and could support further research on evaluation governance.
  2. The study distinguishes missing public responses from partial and substantive responses, provides coding rules and worked examples, and examines alternative counting units and within-report clustering. These choices improve interpretability and address concerns that the headline result merely reflects how findings are split across reports. The explicit limitation to public evidence also helps constrain the conclusions.
Critical weaknesses
  1. Of the 190 significant-risk findings, 114 are coded as having no located public company response. Although search logs and author review are provided, no blinded re-search estimates how often qualifying public responses were missed. Severity-label agreement does not validate response retrieval. Consequently, the uncertainty surrounding the headline 80% shortfall remains insufficiently characterized, directly affecting confidence in the main empirical contribution.
  2. Findings published across several years share a single response-search cutoff, giving recent findings substantially less time to receive a documented response. Figure F.2 describes delays among located responses but does not resolve this issue for findings without responses. Without a minimum follow-up requirement or analyses using comparable observation windows, the estimated shortfall may be inflated, and the pre-/post-deployment comparison remains difficult to interpret.
Questions for authors
  1. Table B.1 reports 119 logs listing the sources searched across 233 trackable findings. How many of the 114 significant-risk findings classified as having no located response have documented coverage of all five required source classes? Please clarify whether author review of these negative classifications involved fresh searches or examination of the agents’ evidence and logs.
  2. How many significant-risk findings were published at least 30 and 90 days before the search cutoff? What are the shortfall rates within these subsets, and does the pre-/post-deployment difference persist when findings have comparable observation windows? These results would materially affect whether the reported gap can be interpreted as a persistent absence of documented response.
1 comment

Thank you for taking the time to review our paper and share your feedback. We address your comments and questions below.

Question1/Weakness1: Documentation of no-response classifications

Table B.1 reports 119 logs listing the sources searched across 233 trackable findings. How many of the 114 significant-risk findings classified as having no located response have documented coverage of all five required source classes? Please clarify whether author review of these negative classifications involved fresh searches or examination of the agents’ evidence and logs.

All 113 significant-risk findings have no located company response with explicitly documenting search across all five required source classes, in the revised dataset. Our protocol requires search across all five classes, but the logs vary in how explicitly they document them. The original Table B.1 has 119 logs that listed searched sources, but the revised Table B.1 contains 220 of 234 trackable findings that have a dated company-response search log.

Action 1.1: In a fresh blind re-search of all 233 trackable findings, we identified two additional company acknowledgments but no additional documented mitigation. The headline result (82.7% gap) by the independent re-search remains unaffected.

Action 1.2: We have added the updated analysis in Appendix B.2.

Question2/Weakness2: Unequal follow-up periods

How many significant-risk findings were published at least 30 and 90 days before the search cutoff? What are the shortfall rates within these subsets, and does the pre-/post-deployment difference persist when findings have comparable observation windows? These results would materially affect whether the reported gap can be interpreted as a persistent absence of documented response.

We found the overall proportionality shortfall remained high with longer follow-up. Of the 176 significant-risk findings with at least 30 days of follow-up, 146 (83.0%) lacked a proportionate response. Of the 149 findings with at least 90 days, 127 (85.2%) lacked a proportionate response. The overall shortfall was 158 of 191 (82.7%).

Pre-/post-deployment comparison. Yes, the difference reported in Section 4.3 persists when both groups have the same observation window. By day 30, 51.2% of pre-deployment findings and 1.6% of post-deployment findings had a documented substantive response. By day 90, the rates were 48.5% and 1.9%, respectively.

Action 2.1: We have updated appendix F.3 (“Timing and follow-up”) to report both analyses. It now gives the counts and explains how we counted responses already public at disclosure.

Question 2.2: If the new analysis addresses your concerns, we’d appreciate reconsidering of the overall assessment in the light of revision. If either concern remains, we’d be happy to clarify.

Reviewer_2DyNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether third-party evaluations of frontier AI systems lead to publicly documented responses. Using LLM-assisted extraction and coding with author review, the authors assemble 1146 findings from 457 reports. They report that 152 of 190 trackable significant-risk findings lack a substantive, publicly documented developer response. They also compare the evaluation ecosystem with sectoral oversight bodies.

Critical strengths
  1. The research question of whether published third-party evaluation findings lead to publicly documented developer responses is important. Studying individual findings is a good choice because findings within one report may warrant different responses. Keeping trackability, severity, and response strength separate makes the response assessment interpretable, while explicit evidence rules make individual judgments inspectable.

  2. The main result is that 152 of 190 trackable significant-risk findings (80%) lack a substantive, publicly documented developer response. The shortfall remains high across the alternative counting units and populations in Table F.1, reducing concern that the result depends on one counting convention. The authors also carefully distinguish missing public documentation from company inaction, keeping their interpretation within what the public record can support.

Critical weaknesses
  1. The proportionality standard is not sufficiently justified.

Table 4 requires a Substantive response for every significant-risk finding, although appropriate remedies may differ across risks. It is unclear what action would satisfy this rule for the evaluation-awareness example in Table 3. The choice matters: the counts in Appendix G give a 69% shortfall if partial action is sufficient and 60% if acknowledgment also counts. These thresholds measure different outcomes. I would like a clearer explanation of why the chosen standard captures proportionate follow-up across the findings being studied.

  1. The empirical response analysis requires stronger validation of its measurements and comparisons.

The accuracy of the coding and search procedures is not independently established, and concrete classification and attribution errors remain. Section 3.4 says human reviewers saw all three model votes and rationales, which may anchor their judgments, and agreement before adjudication is not reported. SHANGHAIAILAB-2025-02-JAI2 is an empirical Claude jailbreak result classified as non-empirical, while Table F.4 associates Google with three responses made by OpenAI. The supplement also excludes "ordinary accuracy" findings without a general exclusion rule in the paper. For response retrieval, Section 7 acknowledges that recall was not estimated. A blinded expert audit and independent re-searches would help establish the accuracy of these measurements.

The comparisons do not adequately distinguish coordinated disclosure from later follow-up. Section 4.3 already recognizes publication timing. Restricting the released data to reports hosted outside developer websites gives substantive responses for 9 of 16 pre-deployment significant-risk findings versus 8 of 125 post-deployment findings. However, seven of those nine pre-deployment responses were already public when the report appeared. Developer engagement and unequal observation periods therefore need more direct treatment. Missing data also affect the public-coverage comparison in Appendix F.2: 19 of the 32 findings without documented coverage have all four public-coverage columns (media, academic, social, verbatim) blank (23 if only the three fields counted in Table B.1 are used). These records should be separated from completed negative searches.

Questions for authors
  • What action would count as proportionate for the evaluation-awareness finding in Table 3?
  • What was the agreement between the two reviewers before adjudication?
  • Can you provide a blinded expert audit of trackability, severity, and developer attribution on a stratified sample?
  • Can you independently re-search a stratified sample to assess how often responses were missed?
  • What general rule determines the supplement’s "ordinary accuracy" exclusions?
  • Can you separate responses already public at report publication from subsequent responses, then account for unequal follow-up periods?
3 comments

Thank you for taking the time to review our paper and share your feedback. We address your comments and questions below.

Question1/Weakness1 – “proportionate” responses

What action would count as proportionate for the evaluation-awareness finding in Table 3?

Clarifying the definition: Generally, larger risks should demand more substantive responses from AI companies. This motivated us to code responses as "proportionate" or not. Per Table 4, "proportionate" means at least a "partial" response for a low-risk finding and at least a "substantive response with explicit attribution" response for a "significant" risk finding. "Evaluation awareness" is classified as a "significant risk," so "proportionality" requires a "significant" response. "Proportionality" is meant to reflect a notion of "appropriateness," and is a coding category that we designed to be contextually assessed – it does not require any specific remedy for any specific risks. For example, Anthropic could meet the standard by publicly documenting a specific change that directly addresses the behavior UK AISI identified and explicitly linking that change to the finding. We coded the response in Table 3 as Acknowledged because Anthropic reproduced the measurement without documenting such a change.

Action 1.1: We have added a glossary to the appendix.

Question2/Weakness2 – Accuracy of coding procedure

What was the agreement between the two reviewers before adjudication?

Grading, rather than independent comparison, is the gold standard for assessing AI-labels on long documents. Based on a great deal of research on human-AI collaboration in long document labeling, we opted for a grading approach rather than comparison to independent human labels. Anchoring is a possible pitfall, but pure human error when annotating large numbers of papers would be a pitfall with the alternative.

We have established checks in our methodology. Three frontier models from different providers produced initial severity labels using our iteratively designed prompt. The final severity label was derived from taking the majority of 3 models with human review and agreement (Section 3.4; Appendix C.1). Human review overturned the provisional severity label for 2 of the 234 “trackable evaluation findings” for our final annotation approach. This lower-bounds inter-author disagreement. The process of severity labelling involved checking source evidence, agreement of authors, manual prompt design, and author overturn of some severity labels.

Action 2.1: We have added an explanation of this validation choice in Appendix C.1

Question3/Weakness2 – Blinded audit

Can you provide a blinded expert audit of trackability, severity, and developer attribution on a stratified sample?

We did an internal blinded audit. We investigated this concern by reviewing all potentially empirical findings classified as non-empirical and every developer assignment underlying Table F.4. We found and corrected the overlapping Shanghai records, separated developer-specific findings, removed six company-internal findings, and added two Irregular findings. These corrections are reflected in Sections 3.2-3.4 and Appendix D.

Action 3.1: For developer attribution, we rebuilt Table F.4 in Appendix F.3. We made sure that each finding is assigned to one accountable developer and each response is credited only to the responding company. This resolved the reviewer’s Weakness no. 2 regarding Google/OpenAI from Appendix F.

Action 3.2: We have revised our dataset after these corrections, including all charts and statistics, and now the paper has 1143 total findings from 456 reports, including 234 trackable findings. Of 191 significant-risk trackable findings, 158 (82.7%) lack a proportionate publicly documented response, compared to 80% in original submission.

Question4/Weakness2 – Re-searching

Can you independently re-search a stratified sample to assess how often responses were missed?

Action 4.1: We found two new responses. We conducted a fresh blind re-search of all 233 “trackable evaluation findings” and found two qualifying company responses among the total of 145 findings, previously coded as having no located response. The re-search corresponds to an observed miss rate of 1.4%. Both response documents acknowledge the relevant findings, but did not document a specific mitigation. The headline result by the independent re-search is unaffected.

Action 4.2: We have added the re-search details and results in Section B.2

Question5/Weakness2 – “Ordinary accuracy”

What general rule determines the supplement’s "ordinary accuracy" exclusions?

This was a coding label that we did not use in the paper or appendix; it only appeared in our supplemental upload: “Ordinary accuracy” was a coding label used for corpus construction initially, which was later replaced by well-defined scope and eligibility rules described in Section 3.2, Appendix A.1, A.2. The term referred to findings that are related to general task-performance failures, such as factual errors, hallucinations, benchmark correctness, or generic reliability, without identifying any distinct safety-relevant consequence or failure. This was replaced in our analysis, and it is not present anywhere in the main paper and appendix, but only in the supplementary logs.

Question6/Weakness2 - “Response and unequal follow-up periods”

Can you separate responses already public at report publication from subsequent responses, then account for unequal follow-up periods?

Action 6.1: We analysed the responses published before, on the same day, and after the findings were published. Among the 78 significant-risk findings with a located response, 10 responses were published before the report, 42 were published on the same day, and 26 after the report. This timing is consistent with coordinated disclosure, rather than necessarily showing company action taken after publication.

Action 6.2: We have updated Section 4.3 with significant-risk breakdown. Appendix Figure F.2 shows the timing distribution for all 88 dated responses.

Action 6.3: We analyze the findings with at least 30, 90, and 180 days of follow-up. The proportionality shortfall was 83.0%, 85.2%, and 89.2%, respectively, compared with 82.7% overall. We found that the proportionality gap remains high when the analysis is restricted to findings with longer follow-up periods.

Action 6.4: We have added the minimum follow-up results and the fixed-window comparison in Appendix F.3.

Question: If the new analysis addresses your concerns, we’d appreciate reconsidering of the overall assessment in the light of revision. If either concern remains, we’d be happy to clarify.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised changes to the paper

The revised PDF includes four main changes.

  • Dataset and results (Sections 3.2–3.4 and 4.4, Appendix F.3). We ellaboratively defined which findings were included and how they were assigned to developers, then updated the tables and figures. In the revised dataset, 158 of 191 significant-risk findings (82.7%) lacked a proportionate, publicly documented company response.
  • Response searches (Appendix B.2). We clarified what the search logs document and reported the results of a fresh blind search. It found two additional company acknowledgments, neither of which documented a mitigation.
  • Follow-up time (Section 4.3 and Appendix F.3). We separated responses already public when a finding appeared from later responses. The overall shortfall remained high with longer follow-up, and pre-deployment findings still had higher substantive response rates when both groups had the same 30- or 90-day window.
  • Definitions and oversight (Glossary, Section 5, and Appendix H). We added a glossary and described how an independent body could track findings, set response deadlines, and record whether fixes were independently verified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.