acceptodds
Under review as a conference paper at ICLR 2027

Beyond Frontier Evaluation: Auditable Energy, Affordability, and Data-Governance Metrics for AI Deployed in the Global South

Abstract

AI evaluation and governance are indexed to frontier models, yet most people meet AI through distilled, fine-tuned, or platform-embedded systems, often in low- and middle-income countries where rules written for frontier labs propagate weakly. We argue that evaluation should target diffusion capacity in the technology-policy sense: whether a deployed system is affordable, runs on local infrastructure, and can be governed where it lands. We operationalize it with three instruments an auditor can compute or verify within one jurisdiction from public evidence. Joules per Task (JpT) labels inference energy with its measurement boundary and task harness, since rankings can reverse when either changes; in a provenance-preserving collation, published per-prompt figures for text generation (measured and estimated) span nearly three orders of magnitude, and boundary choice alone moves one provider's median-prompt estimate by 2.4×. A Price Accessibility Index (PAI) relates price to income: on a 20-economy panel, a USD 20 monthly subscription exceeds the UN 2%-of-monthly-income affordability line for broadband in 19 economies, burdens range from 6.5× to about 82× the US level, and a purchasing-power discount alone leaves half the panel above the line. A data-value-retention (DVR) vector grades disclosure of onshore storage, consent, local revenue sharing, and portability. Verdict rules grade each axis separately, withhold positive verdicts below a host-country governance threshold, and let oversight and rights failures cap a grade rather than be averaged away; by construction they are monotone in adverse evidence and admit no aggregate that can offset a veto, and each identified gaming route has a closing rule. On three focal cases (an open-weight model release, a data-centre build-out, and a mobile-money platform), the rules return capped and ungraded verdicts alongside positive ones; a re-verified dataset of 92 documented engagements by 39 Chinese AI firm groups in 24 countries grounds the cases descriptively. We keep in-sample consistency checks separate from a forward test specified in advance, and release the dataset, code, and JpT task suite.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI evaluation, deployment auditing, inference energy, measurement boundaries, AI affordability, data governance, open-weight models, Goodhart's law, AI governance, Global South

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3tE5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that AI governance should target within-jurisdiction evaluations and proposes an analysis framework to instantiate its arguments. Specifically, instead of focusing on the evaluation on frontier models, researchers should also evaluate the specific situations in the local area of interest (e.g., Jakarta) such as the affordability (PAI) and management by local authority (DVR). Authors have also found that the Joules per Task (JpT) reported by different measurements can have significant differences. Case studies demonstrate the proposed analysis framework.

Critical strengths
  1. The proposed argument is generally clear and seems an underrepresented topic in recent discussions on AI governance.
  2. The anlyses (Figures 1 and 2) are concrete and useful to this research direction.
Critical weaknesses
  1. This paper is hard to follow. Certain linguistic patterns produce a lot of complex sentences such as "....: ..., ..., and ..." or "...: ...; ...", which undermine the readability. Though main arguments and figures are understandable, the low readability may prevent readers from engaging in deeper thinking.
  2. Although this paper's main statement is regionality of AI governance, analysis around Figure 1 seems not directly related to that discussion. In my understanding, Figure 1 says that different measures can lead to very different estimates on energy consumption. Why this figure helps or instantiates regional analysis? Please let me know if I am wrong. Additionally, deep discussions around why this argument was valuable and underexplored are also lacking from my perspective. Such discussions are critical given that this paper is close to a position paper.
Questions for authors
  1. How does Figure 1 relate to regionality of AI governance?
  2. I think there is a subtle difference between "AI governance in global south" and "AI governance in a specific region". Both are mixed in this paper. For example, you can also discuss the affordability and local management in Germany or Japan. Your proposed idea and instantiated framework can work on these countries as well, but global south was set as a keyword for this paper. Do you intend to distinguish between the AI governance in global south and in a specific region? Which one is your main argument?
Reviewer_Xns5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper points out that existing AI evaluation frameworks emphasize too much on frontier model capabilities and overlook the practical cost and conditions for them to be deployed in the global south (i.e. low and middle income countries). Then the authors propose three metrics for evaluating diffusion capacity: Joules per Task (JpT), Price Accessibility Index (PAI), and Data-Value Retention (DVR). Furthermore, the paper introduces per-axis grading, jurisdictional governance gates, and veto rules to further tune the raw evaluation results to make them more credible and fair. The framework is illustrated along with an affordability analysis covering 20 economies, a dataset of 92 documented corporate engagements, and supplemented with 3 case studies to demonstrate its application.

Critical strengths

(1) Underexplored, interesting and important topic. The paper addresses an interesting, important, and underexplored aspect of AI evaluation. AI, as a general productivity tool, should benefit the entire mankind rather than a few monopolies only in developed countries. The paper targets the deployment of AI in underdeveloped countries and focuses on AI for the global south, which is very meaningful and should be encouraged.

(2) The emphasis on auditability and evidence transparency is valuable. Targeting the reality that there is a lot of benchmaxxing, reward hacking, or simply irreproducible hypes in AI performance reporting, the emphasis on auditable evidence and transparency is very valuable, and should be emphasized for AI advances to be fairly recognized.

Critical weaknesses

(1) The main weakness is little to no technical contribution. Although the proposed metrics are great and I fully support from my personal opinion, they look like a position paper rather than a technical paper. There is little to no technical contents in the paper, and thus I doubt whether the paper is in scope.

(2) The proposed metrics are coarse-grained and too idealistic. This point echoes Weakness 1. It is fine to sketch the solutions and metrics as a position paper, but as a technical paper, a practical and justified math definition for the score is expected. It requires justification to use the analogy of broadband on AI. Also, standing in the shoes of for-profit AI companies, the computing cost on their side is not negligible, especially given that AI is not yet a high-profit industry, which is fundamentally different from infrastructure construction in the global south.

Questions for authors

I would be willing to revisit my assessment if the authors could decently address Q1.

Q1: Could you please provide a justification that this paper is in scope as a technical contribution to ICLR rather than a position paper?

Q2: How do you justify the analogy of broadband on AI, and how do you plan to make the proposed metrics acceptable and adoptable to major AI labs?

Reviewer_NwTzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that AI evaluation should target diffusion capacity rather than frontier capability: whether a deployed system is affordable, runs on local infrastructure, and can be governed in the country where it lands. It proposes three instruments: (i) Joules per Task (JpT): energy per task, (ii) Price Accessibility Index (PAI): 12 × monthly price, (iii) Data-Value Retention (DVR): a four-component disclosure vector about transparency. Three test cases are employed for this taxonomy: DeepSeek-R1 with its API (Nigeria, Cambodia), Huawei's Konza national cloud (Kenya) and OPay (Nigeria).

Critical strengths
  • The provenance discipline is careful and useful. Each metrics source's boundary and measured/estimate label instead of normalizing them away, and the paper is honest that the three-order spread describes the published record, not true energy.
  • timely and important topic to take a look beyond mere capability evals, specifically energy consumption and accessibility are getting more and more important
  • good reproducibility, scores and input data available.
Critical weaknesses
  • the paper has virtually no technical nor empirical contribution. this paper is essentially a position paper and rather submitted to the wrong track.
  • while the paper is very open about the metrics boundaries they are only as good as the boundaries. and they move overtime, in its current form this framework is too rigid and needs more flexibility.
  • the evaluation is pretty china-heavy which limits insights and learnings for the community, broader evals would be helpful
Questions for authors

I think this paper is in the wrong track, please reconsider resubmitting the paper to a respective

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.