- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work propose AGC-Bench which integrates 67 benchmarks across six creativity domains and evaluates 83 LLMs. Authors use three LLM judging panel using JRT and trains AGC-Judge (scoring model) on calibrated ratings. PCA identifies a dominant component explaining 82.4% of domain-score variance and suggets general creativity component in LLMs. The paper also investigates reasoning and prompting effects and compares humans and LLMs on five tasks, supported by independent originality ratings from 416 participants.
Critical strengths
- Consolidating fragmented creativity benchmarks into a common evaluation harness is valuable, and domain-level results provide informative model profiles.
- Independent human validation. Source-blinded originality ratings, external story evaluation, and transparent reporting of limitations are important strengths.
Critical weaknesses
(1) Combining existing benchmarks also carries over their limitations. The AI-assisted pipeline makes a large evaluation practical, but it still relies on existing tasks and their original metrics or documented adaptations. If those tasks reward familiar answers, writing style, or task-solving ability rather than creativity, combining them does not necessarily solve these problems. The breadth is valuable, but the paper should explain more clearly what it improves beyond running existing benchmarks in one place. Its ability to measure creativity remains limited by the quality and coverage of the source benchmarks.
(2) The general creativity component may partly reflect that stronger models perform better at most tasks. Finding that models perform well across several creative domains is interesting, but this does not establish a separate creativity ability. AGC-Bench scores are strongly related to general model performance (ρ = .778 with the AA Intelligence Index; Appendix K). Accounting for one reasoning test is helpful, but differences in knowledge, language ability, and instruction following may still explain the pattern. Clearer evidence is needed that the creativity score tells us something beyond which models are generally stronger.
(3) AGC-Judge's contribution to the main findings is unclear. The main benchmark scores and creativity-component analysis use the original metrics and calibrated judge panel; AGC-Judge is trained to approximate that panel. It could make future evaluation cheaper, but its added value to the main scientific findings is less clear. Its agreement with human story ratings is similar to the existing GPT judge (.65 versus .66; Appendix H.7). A direct comparison of evaluation cost, human agreement, and model-ranking accuracy would better establish its practical contribution.
Questions for authors
(1) Beyond bringing existing benchmarks together, which limitations of their tasks or scoring methods does AGC-Bench address? What evidence shows that the combined evaluation measures creativity more reliably?
(2) Among models with similar general performance, does AGC-Bench reveal meaningful differences in creativity that agree with independent human ratings?
(3) What changes when AGC-Judge replaces the calibrated panel in the main evaluation? How much evaluation cost does it save, and how well does it preserve model rankings and agreement with humans?
6 comments
Q: The general creativity component may partly reflect that stronger models perform better at most tasks.
We thank the reviewer for this insightful question, it provided a clear roadmap to rigorously test whether the c-factor is distinct from general capability. As detailed in the Common Response, three new factor analyses prove that c is not a byproduct of model scale, general capability (g-factor), or standard skills (reasoning, knowledge, instruction following):
- Scale-residualized PCA: Removing the effect of model scale (total number of parameters) preserves the dominant component (75.6% residual variance, eigenvalue 4.54 vs. 1.78 noise floor).
- Schmid-Leiman bifactor: Decomposing joint benchmarks (6 creative domains and 6 standard benchmarks) reveals that pure, orthogonal c explains 29.2% of common variance, completely independent of general capability (g-factor), model scale, or specific skills such as instruction following.
- Varimax rotation: Unconstrained PCA naturally separates creativity (36.1% variance) and capability (38.2%) into two equal pillars with near-zero cross-loadings (< 0.09).
Therefore, we conclude that c-factor is not a subordinate byproduct. It stands as a co-dominant, equal-strength pillar of model capability.
Q: AGC-Judge's contribution to the main findings is unclear.
We appreciate this question regarding the distinct roles of the calibrated panel and AGC-Judge. We deliberately decoupled the scientific discovery of the benchmark from the practical deployment tool:
- Why the paper's findings use the calibrated panel: To establish the primary scientific finding (the existence of general creativity factor c across 83 models), psychometric rigor required maximum independence across multiple model families. Using a three-vendor panel (Google, OpenAI, xAI) calibrated with Judge Response Theory (JRT) guaranteed that the c-factor was not an artifact of a single model's weights.
- Why AGC-Judge was developed: Expecting the research community to query three commercial APIs for every evaluation, pay recurring fees, and fit Bayesian IRT models is cost-prohibitive. Distilling the panel into an open-weight model provides a fast, cheap, reproducible drop-in evaluation tool.
Below is the direct comparison of evaluation cost, model-ranking accuracy, and human agreement:
| Evaluation Dimension | Calibrated 3-Judge Panel (Gold Baseline) | AGC-Judge (Open-Weight Qwen3-30B LoRA) | Practical Impact & Paper Location |
|---|---|---|---|
| Primary Purpose | High-fidelity scientific measurement for discovering the c-factor. | Fast, accessible, drop-in evaluation tool for the research community. | Section 3 & 4.2: Decouples scientific discovery from community deployment. |
| Evaluation Cost & Runtime | 2 commercial API queries per item (planned-missing 2-of-3 design; 184k calls across 83 models). • Full 83-model cohort: ≈ 250 to 500 USD (with lightweight mini/flash models) or 1,500+ USD (with full frontier models); takes days due to rate limits. • Evaluating 1 candidate LLM: ≈ 3 to 10 USD. | 1 query per item (50% fewer queries; 92k calls for 83 models, or 1.2k calls per model). • Full 83-model cohort on rented GPU: ≈ 3 to 4 hours on a single A100/H100 (≈ 2.50 USD/hr), costing ≈ 10 to 15 USD total. • Evaluating 1 candidate LLM: ≈ 5 to 10 minutes, costing < 0.30 USD. | Appendix G.5: Slashes evaluation costs by > 95% (< 0.30 USD per candidate LLM vs. commercial APIs). Eliminates rate limits, multi-vendor API dependencies, and model deprecation. |
| Model Ranking Preservation | Defines gold leaderboard composite across 83 models. | Reproduces gold model rankings almost perfectly when replacing the panel entirely. | Figure 2b, Analysis Artifacts: Near-perfect rank preservation on held-out test items (ρ = 0.9765), held-out models (ρ = 0.9758), and held-out benchmarks (ρ = 0.9657). |
| Agreement with Human Raters | Proprietary GPT-4 judge correlates with human gold ratings at ρ = 0.66 on creative stories. | Agrees directly with human gold ratings at ρ = 0.65 (matching proprietary GPT-4) using an open-weight 30B model. | Appendix H.7: Parity with closed frontier LLMs on external human gold ratings (Orwig corpus, 718 stories) without proprietary API dependencies. |
| Source-Preference Bias | Severe self-preference bias: GPT judge favored GPT-4 over humans by +0.83 points (humans rated them equal: +0.01). | Reduces self-preference bias by nearly 40% down to +0.51 points. | Table 10, Appendix H: Significantly fairer across human and AI text than frontier judges, while matching human correlation. |
Conclusion
Replacing the calibrated panel with AGC-Judge cuts queries in half, eliminates recurring API costs, significantly reduces AI self-preference bias, and preserves leaderboard model rankings at ρ ≥ 0.966 to 0.977.
Q: Beyond bringing existing benchmarks together, which limitations of their tasks or scoring methods does AGC-Bench address? What evidence shows that the combined evaluation measures creativity more reliably?
We thank the reviewer for recognizing the value of our pipeline and the breadth of unifying 67 benchmarks. AGC-Bench does not merely run existing benchmarks in one place; we actively engineered multiple layers of quality control and psychometric calibration to solve known limitations of isolated benchmarks.
Below, we summarize the specific limitations addressed and the empirical evidence establishing reliability:
Limitations of Existing Benchmarks Addressed by AGC-Bench
Evidence that the Combined Evaluation is More Reliable
Providing this standardized, open infrastructure is the critical first step that enables the research community to systematically audit, investigate limitations, and improve benchmark tasks.