- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that the admissibility of LLM outputs should be audited before comparing results across models and runs. The paper's primary contribution is empirical, with experiments conducted on a carbon capture, utilization, and storage (CCUS) corpus. The experimental results show that none of the eight clusters satisfies the common-support requirement, rendering the primary estimand non-estimable under the proposed protocol. Based on these findings, the authors recommend auditing output admissibility before reporting cross-system agreement.
Critical strengths
- The paper explicitly distinguishes different output states, including verified zero extraction, abstention, and terminal failure, and demonstrates how their treatment affects the apparent completeness of cross-system measurements.
- The paper provides detailed diagnostic analyses of output failures, which help explain the observed non-estimability under the proposed protocol.
Critical weaknesses
-
The paper presents the observation of non-estimability under a human-designed protocol as its primary empirical contribution. However, although some justification for the protocol design is provided, the methodological rationale for certain design choices and their potential influence on the empirical outcomes remain insufficiently examined. This raises concerns about the extent to which the observed non-estimability depends on the specific protocol settings.
-
The generalizability of the empirical findings is another concern. The experiments are conducted exclusively on a CCUS corpus, which limits the scope of the empirical evidence supporting the paper's broader methodological recommendation regarding admissibility auditing across LLM measurement systems.
Questions for authors
- Could the authors further clarify the motivation for conducting experiments exclusively on the CCUS corpus and provide additional justification for the chosen protocol settings, particularly regarding their potential influence on the empirical findings?