acceptodds
Under review as a conference paper at ICLR 2027

Before LLM Measurements Can Disagree: Auditing Output Admissibility Across Systems

Abstract

Large language models (LLMs) now serve as measurement instruments in socialscience and ML pipelines, and their outputs are compared across models and runs to assess reproducibility. Such comparisons presuppose an admissible output from every call. Yet recent work resamples failed calls or drops them before aggregation, so whether each system produced admissible outputs on the same inputs goes untested. We therefore audit output admissibility before agreement with a locally frozen, stop-rule-governed protocol. It treats every failed call as terminal and keeps verified zero extraction, abstention and terminal failure as separate states. The pipeline first extracts atomic stakeholder claims with verbatim evidence and then classifies them into seven frames. Its inputs are 11 masked documents from 8 carbon capture, utilization and storage (CCUS) project clusters. Three open-weight 7–8B checkpoints were each run in three seeded repeats with fixed prompts and decoding settings. The primary estimand contrasts crosssystem with within-system Jensen–Shannon divergence of cluster-level frame distributions. The frozen protocol requires every cell of every cluster to be complete and stops at the first terminal failure. The run stopped at its first call. We then executed all remaining calls to characterize the failure flow. Only 23 of 153 extraction calls returned schema-valid output with every claim anchored to verbatim evidence. Common support, all 9 cells complete, held in 0 of 8 clusters, so the primary estimand is not estimable. Most failures were evidence misalignments, and even dropping every non-conforming claim let only 2 of 8 clusters pass extraction. Failures recurred across independently seeded repeats even as responses varied, and their kind differed across systems. Token-level replay traced every unparseable constrained response short of the output-length cap to the constraint adapter. Recoding failures as empty extractions or abstentions gave 7 of 8 clusters apparent common support. The study concerns measurement reproducibility for this finite corpus and these checkpoints. We did not validate outputs against human labels, and the study does not assess semantic accuracy, construct validity or which model is correct. For these checkpoints, prompts and decoder, which inputs yielded an admissible output depended on the system. Admissibility should therefore be audited before cross-system agreement is reported.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Output Admissibility, LLM-Based Annotation, Measurement Reproducibility.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_udtqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the admissibility of LLM outputs should be audited before comparing results across models and runs. The paper's primary contribution is empirical, with experiments conducted on a carbon capture, utilization, and storage (CCUS) corpus. The experimental results show that none of the eight clusters satisfies the common-support requirement, rendering the primary estimand non-estimable under the proposed protocol. Based on these findings, the authors recommend auditing output admissibility before reporting cross-system agreement.

Critical strengths
  1. The paper explicitly distinguishes different output states, including verified zero extraction, abstention, and terminal failure, and demonstrates how their treatment affects the apparent completeness of cross-system measurements.
  2. The paper provides detailed diagnostic analyses of output failures, which help explain the observed non-estimability under the proposed protocol.
Critical weaknesses
  1. The paper presents the observation of non-estimability under a human-designed protocol as its primary empirical contribution. However, although some justification for the protocol design is provided, the methodological rationale for certain design choices and their potential influence on the empirical outcomes remain insufficiently examined. This raises concerns about the extent to which the observed non-estimability depends on the specific protocol settings.

  2. The generalizability of the empirical findings is another concern. The experiments are conducted exclusively on a CCUS corpus, which limits the scope of the empirical evidence supporting the paper's broader methodological recommendation regarding admissibility auditing across LLM measurement systems.

Questions for authors
  1. Could the authors further clarify the motivation for conducting experiments exclusively on the CCUS corpus and provide additional justification for the chosen protocol settings, particularly regarding their potential influence on the empirical findings?
Reviewer_qe1WOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper is an empirical evaluation/audit of a carbon capture dataset . A protocol

Critical strengths
  • Well-structured paper
  • Good disclosures of models, licencing, and parameters
  • Motivation appears good.
  • Use of open models
Critical weaknesses

I have a few concerns about this paper. First and foremost, it is very hard to understand what the authors are proposing, and how they are achieving it. The language is dense, imprecise, and needs multiple reviews to get an idea of it. Second, it is primarily an empirical audit of a single dataset. The accompanying claims () and proposed solutions (an auditability protocol) are hard to evaluate in a specific corpus, and much harder to evaluate when said corpus isn't described or released in the main body of the work.

This is called in the abstract as a limitation, but it still consists a major shortcoming of the paper.

Here's a few ways I would improve the paper for a future submission:

  1. The dataset is very narrow for the claims. It is not described at all anywhere in the work (nor it is cited, which suggests that it is proprietary). The claim is on auditability, but this is a carbon capture dataset. It is very unlikely such a generalised conclusion will hold or extend from this work.I would add a few more multi-domain corpora.
    • Likewise, the models are quite old. This is a reproducibility study, so open models is good; but newer models are a must since there have been major changes from (say) Qwen2.5 to Qwen3 (which was out in April 2025) or 3.5 (February 2026).
  2. Results sections are separated from discussion (which is good), but the subsection titles give the conclusion/interpretation. That's what the discussion is for. Keep that section objective.
  3. Lack of human validation does not bode well for the results overall, given the narrow experimental setup.

Additionally:

  1. The abstract needs work. As it stands, it is extremely unclear. There is no need to put all the results in it, either. I appreciate the addition of limitations directly in there, but this can be put in the introduction (and limitations/discussion).
  2. Some things do not belong in a scientific paper. For example, L91-92: 'No part of the protocol was changed after every response was read'. This is to be expected of scientific work.
  3. In a meta-science work (and science in general), statistics are everything. I would advice the authors to improve the experimental protocols.
Questions for authors

How would the results from your corpus extend or apply to other domains?

Reviewer_inL8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether different LLMs can reliably extract different stakeholder info/claims from documents before comparing their results. It finds that most outputs fail basic formatting checks pass, making cross-model agreement very diifficult to measure reliably.

Critical strengths
  • I really like that the paper treats the output admissibility as a prerequisite for cross-model agreement.

  • The paper is transparent - systematic failure analysis across 153 calls, three models, and three repeats, including evidence-alignment diagnostics, decoder-adapter debugging, and sensitivity...

Critical weaknesses
  • I didn't find a primary result in this paper, the proposed cross-model agreement statistic was never computed, was it?

  • One failed call invalidates the entire comparison no? i think this is a self-defeating design...

  • Why we would need this overly rigid validation? A single misaligned quotation or actor mention invalidates all claims in a response.

  • I think technical failures conflated with model failures: 16 failures came from decoder-adapter defects; 21 hit the output-token limit. So, why to confalte?

  • Why no ground truth: All nine valid Qwen extractions were empty, without human verification that no claims existed.

  • The 71 synthetic controls use the first two words of sentences as actor mentions, right, why?

Questions for authors

My questions are listed in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.