acceptodds
Under review as a conference paper at ICLR 2027

AGC-Bench: Measuring Artificial General Creativity

Abstract

Creativity research has long debated whether creativity is a general ability or specific to particular domains (e.g., creative writing, visual art, scientific discovery), and whether it is separable from general intelligence. Both questions now apply to LLMs, but a fragmented evaluation landscape across hundreds of heterogeneous creativity benchmarks has left them empirically intractable. We introduce AGC-Bench, a meta-benchmark for artificial general creativity built from a PRISMA-compliant systematic review of the AI creativity literature ( papers screened, unique benchmarks identified) paired with an agentic onboarding harness that converts source-paper benchmarks into runnable HELM-style scenarios. The first release covers datasets ( text-only, multimodal) spanning brainstorming, problem solving, STEM, narrative, figurative language, and humor. To address bias in LLM-as-judge, we apply Judge Response Theory (JRT)—a psychometric calibration of judge leniency/severity—with three frontier LLM judges. We then fine-tune Qwen3-30B-A3B-Instruct-2507 on the resulting JRT-corrected ratings to produce AGC-Judge, an open-weight scoring model that matches the three-judge ensemble and predicts frontier-judge ratings with high accuracy on creativity benchmarks it was not trained on. Results reveal frontier models at the top of the leaderboard, with open-weight models close behind. However, different models show different creative strengths, ranking higher on some domains (e.g., creative writing) than others (e.g., scientific ideation). We conduct several experiments and report three main findings. First, applying factor analysis across LLMs, we recover a single creativity factor ‘c’, analogous to the ‘g’ factor of general intelligence, that explains % of variance, related to but separable from general knowledge and reasoning. Second, in within-model comparisons, we show that prompting models to “be creative” boosts their performance far more than enabling their reasoning capability, evidence that the benchmark tracks creativity specifically. Third, on a human-matched subset of five tasks, the top human still leads the top LLM on the novelty and diversity composite. We release AGC-Bench with a public leaderboard, the onboarding harness, AGC-Judge weights, and human comparison data as open infrastructure for measuring AI creativity at scale.

Primary area:
datasets and benchmarks
Keywords:
creativity, benchmarks, evaluation, large language models, psychometrics, item response theory, LLM-as-a-judge, human-AI comparison, human evaluation, open-weight models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4hXoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work propose AGC-Bench which integrates 67 benchmarks across six creativity domains and evaluates 83 LLMs. Authors use three LLM judging panel using JRT and trains AGC-Judge (scoring model) on calibrated ratings. PCA identifies a dominant component explaining 82.4% of domain-score variance and suggets general creativity component in LLMs. The paper also investigates reasoning and prompting effects and compares humans and LLMs on five tasks, supported by independent originality ratings from 416 participants.

Critical strengths
  • Consolidating fragmented creativity benchmarks into a common evaluation harness is valuable, and domain-level results provide informative model profiles.
  • Independent human validation. Source-blinded originality ratings, external story evaluation, and transparent reporting of limitations are important strengths.
Critical weaknesses

(1) Combining existing benchmarks also carries over their limitations. The AI-assisted pipeline makes a large evaluation practical, but it still relies on existing tasks and their original metrics or documented adaptations. If those tasks reward familiar answers, writing style, or task-solving ability rather than creativity, combining them does not necessarily solve these problems. The breadth is valuable, but the paper should explain more clearly what it improves beyond running existing benchmarks in one place. Its ability to measure creativity remains limited by the quality and coverage of the source benchmarks.

(2) The general creativity component may partly reflect that stronger models perform better at most tasks. Finding that models perform well across several creative domains is interesting, but this does not establish a separate creativity ability. AGC-Bench scores are strongly related to general model performance (ρ = .778 with the AA Intelligence Index; Appendix K). Accounting for one reasoning test is helpful, but differences in knowledge, language ability, and instruction following may still explain the pattern. Clearer evidence is needed that the creativity score tells us something beyond which models are generally stronger.

(3) AGC-Judge's contribution to the main findings is unclear. The main benchmark scores and creativity-component analysis use the original metrics and calibrated judge panel; AGC-Judge is trained to approximate that panel. It could make future evaluation cheaper, but its added value to the main scientific findings is less clear. Its agreement with human story ratings is similar to the existing GPT judge (.65 versus .66; Appendix H.7). A direct comparison of evaluation cost, human agreement, and model-ranking accuracy would better establish its practical contribution.

Questions for authors

(1) Beyond bringing existing benchmarks together, which limitations of their tasks or scoring methods does AGC-Bench address? What evidence shows that the combined evaluation measures creativity more reliably?

(2) Among models with similar general performance, does AGC-Bench reveal meaningful differences in creativity that agree with independent human ratings?

(3) What changes when AGC-Judge replaces the calibrated panel in the main evaluation? How much evaluation cost does it save, and how well does it preserve model rankings and agreement with humans?

6 comments
AuthorsOpenReview ↗

Q: Beyond bringing existing benchmarks together, which limitations of their tasks or scoring methods does AGC-Bench address? What evidence shows that the combined evaluation measures creativity more reliably?

We thank the reviewer for recognizing the value of our pipeline and the breadth of unifying 67 benchmarks. AGC-Bench does not merely run existing benchmarks in one place; we actively engineered multiple layers of quality control and psychometric calibration to solve known limitations of isolated benchmarks.

Below, we summarize the specific limitations addressed and the empirical evidence establishing reliability:

Limitations of Existing Benchmarks Addressed by AGC-Bench

Quality ControlOur ContributionLocation & Exact Results
Judge bias & scale inconsistencyPrior benchmarks commonly rely on uncalibrated single judges, keeping judge-bias unresolved. We use a 3-judge panel calibrated via Judge Response Theory (JRT) to remove judge-specific severity. This ensures all LLMs are judged on a standardized and de-biased scale.Section 3, Appendix G: Bayesian Graded Response Model on 184,160 ratings (24 benchmarks) detects and removes judge severity drift (ranging from β = -0.90 for GPT-4.1-mini to β = +0.04 for Gemini-3-Flash) to yield calibrated ability scores θ.
Silent bugs & parser errorsExisting benchmark scripts often fail silently on parse errors. We audited and fixed parsing code to eliminate silent zero-scores and stray number extractions.Appendix G.1: Code audit across 35 benchmark evaluators patched naive integer extractors in 23 scripts, reducing parser-induced error/zero contamination from 0.32% down to 0.00%.
Verbosity & style confoundsIndividual benchmarks rarely test whether models win via sheer length or superficial polish. We control for word count and test style invariance holding ideas constant.Section 4.4, Appendix L: Regression partialling out log(word count) confirms creativity prompt gains remain massive (d_z = +1.40). Counterfactual style-flip probe on 50 paired cells shows negligible score shift when swapping human vs. AI styles (d ≤ 0.29).

Evidence that the Combined Evaluation is More Reliable

Reliability DimensionOur ContributionLocation & Exact Results
Task-specific noise cancellationSingle benchmarks contain idiosyncratic noise. Aggregating 67 standardized scenarios allows task quirks to cancel out, yielding stable model rankings.Table 12, Appendix J: Scale achieves high reliability (α = 0.957). 1,000 draws of 50% subsampling yield median rank correlation ρ = 0.986. Balancing all domains to k = 3 tasks (18 tasks, 100 seeds) preserves the dominant factor (λ₁ = 3.864, 63.6% variance) and rank stability (ρ = 0.9596).
Alignment with human ratersWe validate that our automated benchmark scores and AGC-Judge align with independent, source-blinded human originality ratings.Appendix L.1, Appendix H: Multilevel ordinal modeling on 89,983 blind ratings (416 Prolific raters) correlates with AGC-Bench at ρ = 0.646 across 80 LLMs. On 718 external stories, AGC-Judge matches a frontier GPT judge against human gold (ρ = 0.65 vs. 0.66).
Generalization to unseen tasksPrior evaluators are restricted to single tasks. We train open-weight AGC-Judge across 24 calibrated benchmarks to learn generalizable scoring criteria.Section 4.2, Figure 2b: Evaluation on 8,803 responses across 3 held-out benchmarks shows AGC-Judge predicts panel consensus at pooled Spearman ρ = 0.83.

Providing this standardized, open infrastructure is the critical first step that enables the research community to systematically audit, investigate limitations, and improve benchmark tasks.

AuthorsOpenReview ↗

Q: The general creativity component may partly reflect that stronger models perform better at most tasks.

We thank the reviewer for this insightful question, it provided a clear roadmap to rigorously test whether the c-factor is distinct from general capability. As detailed in the Common Response, three new factor analyses prove that c is not a byproduct of model scale, general capability (g-factor), or standard skills (reasoning, knowledge, instruction following):

  • Scale-residualized PCA: Removing the effect of model scale (total number of parameters) preserves the dominant component (75.6% residual variance, eigenvalue 4.54 vs. 1.78 noise floor).
  • Schmid-Leiman bifactor: Decomposing joint benchmarks (6 creative domains and 6 standard benchmarks) reveals that pure, orthogonal c explains 29.2% of common variance, completely independent of general capability (g-factor), model scale, or specific skills such as instruction following.
  • Varimax rotation: Unconstrained PCA naturally separates creativity (36.1% variance) and capability (38.2%) into two equal pillars with near-zero cross-loadings (< 0.09).

Therefore, we conclude that c-factor is not a subordinate byproduct. It stands as a co-dominant, equal-strength pillar of model capability.

AuthorsOpenReview ↗

Q: AGC-Judge's contribution to the main findings is unclear.

We appreciate this question regarding the distinct roles of the calibrated panel and AGC-Judge. We deliberately decoupled the scientific discovery of the benchmark from the practical deployment tool:

  • Why the paper's findings use the calibrated panel: To establish the primary scientific finding (the existence of general creativity factor c across 83 models), psychometric rigor required maximum independence across multiple model families. Using a three-vendor panel (Google, OpenAI, xAI) calibrated with Judge Response Theory (JRT) guaranteed that the c-factor was not an artifact of a single model's weights.
  • Why AGC-Judge was developed: Expecting the research community to query three commercial APIs for every evaluation, pay recurring fees, and fit Bayesian IRT models is cost-prohibitive. Distilling the panel into an open-weight model provides a fast, cheap, reproducible drop-in evaluation tool.

Below is the direct comparison of evaluation cost, model-ranking accuracy, and human agreement:


Evaluation DimensionCalibrated 3-Judge Panel (Gold Baseline)AGC-Judge (Open-Weight Qwen3-30B LoRA)Practical Impact & Paper Location
Primary PurposeHigh-fidelity scientific measurement for discovering the c-factor.Fast, accessible, drop-in evaluation tool for the research community.Section 3 & 4.2: Decouples scientific discovery from community deployment.
Evaluation Cost & Runtime2 commercial API queries per item (planned-missing 2-of-3 design; 184k calls across 83 models). • Full 83-model cohort: ≈ 250 to 500 USD (with lightweight mini/flash models) or 1,500+ USD (with full frontier models); takes days due to rate limits. • Evaluating 1 candidate LLM: ≈ 3 to 10 USD.1 query per item (50% fewer queries; 92k calls for 83 models, or 1.2k calls per model). • Full 83-model cohort on rented GPU: ≈ 3 to 4 hours on a single A100/H100 (≈ 2.50 USD/hr), costing ≈ 10 to 15 USD total. • Evaluating 1 candidate LLM: ≈ 5 to 10 minutes, costing < 0.30 USD.Appendix G.5: Slashes evaluation costs by > 95% (< 0.30 USD per candidate LLM vs. commercial APIs). Eliminates rate limits, multi-vendor API dependencies, and model deprecation.
Model Ranking PreservationDefines gold leaderboard composite across 83 models.Reproduces gold model rankings almost perfectly when replacing the panel entirely.Figure 2b, Analysis Artifacts: Near-perfect rank preservation on held-out test items (ρ = 0.9765), held-out models (ρ = 0.9758), and held-out benchmarks (ρ = 0.9657).
Agreement with Human RatersProprietary GPT-4 judge correlates with human gold ratings at ρ = 0.66 on creative stories.Agrees directly with human gold ratings at ρ = 0.65 (matching proprietary GPT-4) using an open-weight 30B model.Appendix H.7: Parity with closed frontier LLMs on external human gold ratings (Orwig corpus, 718 stories) without proprietary API dependencies.
Source-Preference BiasSevere self-preference bias: GPT judge favored GPT-4 over humans by +0.83 points (humans rated them equal: +0.01).Reduces self-preference bias by nearly 40% down to +0.51 points.Table 10, Appendix H: Significantly fairer across human and AI text than frontier judges, while matching human correlation.

Conclusion

Replacing the calibrated panel with AGC-Judge cuts queries in half, eliminates recurring API costs, significantly reduces AI self-preference bias, and preserves leaderboard model rankings at ρ ≥ 0.966 to 0.977.

Reviewer_baFJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AGC-Bench, which aims to systematically investigate whether large language models (LLMs) possess general, cross-domain creativity by integrating existing datasets for creativity evaluation. The authors categorize 67 text-based benchmarks into six distinct domains and evaluate 83 models. To address scoring biases among different LLM-based evaluators, they employ Judge Response Theory (JRT) for calibration and subsequently train an open-source evaluation model, AGC-Judge. Experimental results reveal that 82.4% of the variance in model performance across these six domains can be explained by a single principal component. Furthermore, performance gains driven by creativity-focused prompting are found to be greater than those achieved by enabling reasoning modes. Finally, human vs model comparative experiments indicate that the top-performing human still outperforms the best-performing model.

Critical strengths
  1. The research question is meaningful, with objectives that extend beyond simple task-based rankings. Existing approaches to evaluating creativity often focus on a single type of generative task. This paper attempts to unify six distinct domains and investigates whether creativity is a general, cross-domain ability or a domain-specific one.

  2. The authors account for the systematic biases inherent in the "LLM-as-a-Judge" paradigm. The study employs models from three different providers for scoring, uses the JRT (Joint Rating Theory) model to calibrate for variations in rater leniency or strictness, and subsequently trains "AGC-Judge." The calibrated scores show a Spearman correlation of 0.97 with the raw scores, and AGC-Judge demonstrates a correlation of 0.83 on previously unseen creativity benchmarks, indicating that this evaluation methodology has practical utility.

  3. The paper provides a multi-faceted empirical analysis. Beyond conducting a cross-domain factor analysis, the authors examine the correlations between creativity, reasoning, and knowledge. They also incorporate prompt-based intervention experiments and human-evaluation baselines, ensuring that their conclusions do not rely solely on a single leaderboard.

Critical weaknesses
  1. Significant overlap with existing work, which is neither cited nor discussed in the paper. This is a critical issue that I have noted substantial similarities between this paper and previously published work [1]. Like the present study, [1] investigates the assessment of text-based creativity across domains: it compiles a multi-domain, multi-source dataset and develops a specialized creativity evaluation judge model. Similarly, AGC-Bench integrates existing benchmarks across various creativity domains and introduces "AGC-Judge." There is a clear overlap in research objectives, data integration strategies, and the development of assessment models. However, the paper neither discusses nor cites this work. Thus the authors fail to adequately clarify how their study differs from, builds upon, or contributes beyond this existing research.

  2. The evidence for a "general creativity factor" is insufficient to establish the existence of an independent creativity construct. The authors perform a PCA on the aggregated scores of 83 models across six domains, finding that the first principal component accounts for 82.4% of the variance. However, high cross-domain correlations could also stem from shared factors such as overall model capability, parameter scale, training data, or post-training methodologies. Given that the analysis focuses on different models, a single-factor result cannot definitively prove that creativity exists as an independent, universal latent ability.

  3. The method used to aggregate different benchmarks may influence overall rankings and factor analysis. There is a notable disparity in the number of benchmarks across the six domains; for instance, the "Narrative" domain includes 20 benchmarks, while "Brainstorming" contains only 3. Furthermore, task definitions, scoring scales, and evaluation metrics vary across datasets. Even with normalization, the aggregated results may still be sensitive to the composition of tasks within each domain and the specific metrics employed. The authors need to provide more robust evidence that their overall scores and single-factor conclusions remain consistent regardless of the specific benchmarks selected, the weights assigned to different domains, or the aggregation methods used.

  4. While JRT can be used to calibrate the leniency or strictness of raters, it does not fully resolve issues concerning the validity of the ratings themselves. While JRT can calibrate the scoring tendencies of different judges, it cannot guarantee the correctness of the scoring criteria themselves, nor can it eliminate biases shared across multiple models. For instance, judges might systematically favor responses that are longer, more fluent, or more aligned with common linguistic conventions—characteristics that do not necessarily equate to higher originality. The high correlations reported in this paper primarily support the consistency of the scoring results; they do not, on their own, establish the validity of the measurement.

[1] Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator

Questions for authors

Please see weakness.

10 comments
AuthorsOpenReview ↗

Q: Significant overlap with existing work, which is neither cited nor discussed in the paper.

We thank the reviewer for highlighting Cao et al. (arXiv:2505.19236). We appreciate the opportunity to cite this work and to clarify the fundamental differences in scientific objectives, scope, and methodology between the two works. Below, we provide a detailed table comparing the two studies, highlighting our distinct scope and contributions relative to Cao et al.


Clarification

First and foremost, we identified Cao et al. during our PRISMA systematic literature review, indexed it in our 497-benchmark catalog (benchmark_catalog_497.csv), and integrated its generation task (creatset) as one of the 67 evaluated benchmarks in AGC-Bench. All 83 models in our study were evaluated on CreataSet, and their scores are included in our released data (release_data/wide_model_x_dataset.csv). It was an unintentional editorial oversight on our end not to include this work in the main manuscript. We apologize and will immediately rectify this in the revision.

Key differences

Secondly, while both studies touch on multi-domain creativity and judge modeling, their core research questions, scope, and technical contributions are fundamentally distinct (refer to the following table for detailed comparison). In general, CreataSet focuses on the practical challenge of building an evaluator for creative writing. To bypass costly human annotations, they curate a semi-synthetic Chinese dataset with heuristic weak labels to train a scoring model. In contrast, our primary goal is foundational and psychometric: we investigate whether creative ability in LLMs is domain-specific or general. To provide a comprehensive answer, we build a standardized HELM evaluation platform integrating 67 diverse benchmarks across 6 domains, evaluate 83 models, and demonstrate that creativity in LLMs exhibits a general, cross-domain factor (c-factor).


How AGC-Bench Adds Value Beyond CreataSet?

We now provide the additional value of AGC-Bench beyond CreataSet and a systematic comparison of the differences between the two works.

  • CreataSet is a constituent part of AGC-Bench: CreataSet is focused entirely on Chinese genres. In AGC-Bench, CreataSet is simply 1 of 20 benchmarks in our Story / Narrative domain, and 1 of 67 benchmarks across our 6 broader domains (which also cover scientific ideation, lateral problem solving, humor, figurative language, brainstorming, divergent thinking, etc.).

  • Gold-Standard Psychometrics vs. Heuristic Labels: CrEval trains an evaluator based on the assumption that stronger models or creativity-prompted models are always more creative. AGC-Judge is trained on JRT-adjusted scores from frontier judges.

  • Answering Foundational Scientific Questions: CreataSet provides a dataset and a scoring tool. AGC-Bench provides a public repository for creativity evaluation across 6 domains and an empirical investigation of whether creativity exists as a general latent trait in AI.

AuthorsOpenReview ↗

Comparison between two studies

We provide a more detailed comparative table highlighting how AGC-Bench adds value over Cao et al.


Contribution FeatureCreataSet / CrEval (Cao et al., Jan 2026)AGC-Bench (This Work)
Core Motivation & ObjectiveAddress the cost bottleneck of human evaluation by using heuristic weak labels to train a domain-general creative text evaluator (CrEval).Build a unified, open evaluation platform to investigate foundational psychometric questions: Is AI creativity general or domain-specific? Does an independent c-factor exist?
Benchmark Breadth & DiversitySingle benchmark of 8 literary sub-genres (e.g., modern poetry, ancient poetry, lyrics, prose, short texts).Meta-benchmark uniting 67 text benchmarks spanning 6 diverse domains: Brainstorming, Figurative Language, Humor, Problem Solving, STEM, and Narrative. (CreataSet is 1 of the 67 benchmarks).
Language ScopeExclusively Chinese creative writing.Multilingual, including extensive English and non-English tasks.
Models Evaluated10 models evaluated on creative text generation.83 frontier and open-weight models evaluated across all 67 benchmarks.
Judge Training DataWeak heuristic labels: Assumes larger models and creativity-prompted responses are automatically more creative, filtered by heuristics.Psychometric Bayesian calibration: Uses Judge Response Theory (JRT) on 48,299 ratings from a 3-frontier-judge panel to adjust for rater severity and threshold bias.
Judge Model (CrEval vs. AGC-Judge)CrEval (7B/14B): Pairwise classifier predicting win/tie/loss on response pairs.AGC-Judge: Open-weight model (Qwen-30B LoRA) matching the frontier JRT-calibrated panel, generalizing to unseen benchmarks (ρ = 0.83) and unseen models (ρ = 0.94).
Psychometric DiscoveryNone. Does not investigate factor structure or latent ability constructs.Discovers and validates the general creativity factor (c) via PCA, Horn's parallel analysis, model scale residualization, and Schmid-Leiman bifactor modeling.
Human Validation30 human annotators labeling ~3,000 response pairs to evaluate judge agreement.Independent study with 416 humans across 5 divergent thinking tasks (CAP), proving that LLMs exhibit significantly higher cross-domain generality than humans (p = 0.028).
AuthorsOpenReview ↗

Q: The evidence for a "general creativity factor" is insufficient to establish the existence of an independent creativity construct.

We appreciate this critical observation. As detailed in our common response, we conducted two new factor analyses to rigorously test whether the c-factor is a proxy for model scale or general capability. The results from our analysis reinforce our original finding: LLMs exhibit a domain-general creative capability that spans diverse tasks.

We summarize the findings here and provide detailed discussion in the common response.

  • Independent of Model Scale (Residual PCA): We removed the effect of model scale by regressing out total parameters across the 6 creative domains (N = 42). Conducting PCA on the residuals shows that the dominant component survives, explaining 75.6% of residual variance (eigenvalue = 4.537 vs. 1.775 parallel analysis noise ceiling, Cronbach’s alpha = 0.935). Hence, model scale does not account for the c-factor.

  • Independent of General Capability (Bifactor Analysis): We jointly modeled the 6 creative domains and 6 standard capability benchmarks (GPQA, HLE, SciCode, IFBench, τ², CritPT) on size-residualized data (N = 34). Schmid-Leiman orthogonalization shows that the specific c-factor accounts for 29.2% of common variance (vs. 29.0% for the standard benchmark specific factor). Because this factor is strictly orthogonal to both the global g-factor and standard benchmarks, this variance represents the isolated contribution of the c-factor.

Reviewer_i7TqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a benchmark from a review of creativity benchmarks. They use the benchmark to score 83 LLMs, calibrates the LLM judges with Judge Response Theory, and distills the judges into an open-weight scorer (AGC-Judge). It then runs PCA over six domain composites and reports a single “creativity factor” C. The authors also compares models with humans on five tasks. Overall, I enjoyed the infrastructural rigor behind the submission, but I'm pretty sure this "C" factor is just general capability of the model.

Critical strengths

This is useful infrastructure! The catalog, harness, judge weights, and human data are all nice infrastructural contributions.

Critical weaknesses

The "C' factor is likely just general capability. Any set of LLM benchmarks across many models yields a dominant first component, and scrutinizing that first component is necessary but lacking in this submission. Parting out a single task doesn’t establish that C is separate from general capability.

Questions for authors

Two simple ones that should be easy to do:

  1. Does C survive a joint factor analysis that includes standard capability benchmarks?

  2. What does the same PCA yield on six "non-creativity" benchmarks?

3 comments
AuthorsOpenReview ↗

Responses to the two questions

We thank the reviewer for these constructive and insightful questions. They directly challenged our initial interpretation and provided a clear roadmap to rigorously test whether the c-factor is distinct from general capability.

As detailed in the common response, we conducted these analyses on residualized measurements from open-weight models (N = 34), ensuring all findings are free from the confounding effect of model size. Below are the direct answers to your two questions.


Q1: Does C survive a joint factor analysis that includes standard capability benchmarks?

Yes, the c-factor clearly survives. We tested this using two complementary methods across the joint set of 6 creative domains and 6 standard capability benchmarks (GPQA, HLE, SciCode, IFBench, τ²-Bench, CritPT):

  1. Schmid-Leiman Bifactor Analysis: We decomposed the joint system into three mutually orthogonal components: a global capability factor (g-factor), a specific creativity factor (c-factor), and a specific capability factor (only for the standard benchmarks). The global capability factor explains 41.7% and the specific capability factor explains 29.0% of the common variance. On the other hand, the specific c-factor explains 29.2% of common variance. Because these components are mathematically orthogonal, this 29.2% of variance contains zero overlap with model scale, general capability, or skills like reasoning, coding, tool usage, instruction following etc.

  2. Unconstrained Exploratory PCA with Varimax Rotation: To ensure the separation was not an artifact of the constraints enforced by the Schmid-Leiman analysis, we conducted unconstrained PCA. We extracted two components from the residualized matrix and applied Varimax rotation to investigate if separation between creativity and standard benchmarks naturally emerge. A clean double dissociation emerged naturally:

  • The creativity component loads heavily on creative domains (0.70 to 0.93) and has near-zero cross-loadings on standard benchmarks (0.08 on IFBench, 0.09 on GPQA).
  • The capability component loads heavily on standard benchmarks (0.76 to 0.90) and has near-zero cross-loadings on creative tasks (0.01 on Humor, 0.01 on Problem Solving).
  • The two components account for 36.1% and 38.2% of total variance, respectively.

Therefore, we conclude that c-factor is not a subordinate byproduct. It stands as a co-dominant, equal-strength pillar of model capability.


Q2: What does the same PCA yield on six "non-creativity" benchmarks?

Running the identical PCA on the 6 standard capability benchmarks confirms the reviewer's intuition: they independently form a dominant g-factor. The first principal component explains 72.7% of the residual variance (82.7% unadjusted), with an eigenvalue of 4.362 (4.959 unadjusted, Cronbach’s alpha = 0.923). In parallel analysis over 1,000 random matrices, the 95th percentile noise ceiling is 1.863. Because 4.362 is well above noise, standard benchmarks exhibit their own unidimensional structure. Crucially, as shown in Q1, when evaluated jointly, this capability factor separates cleanly from the creativity factor.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

As the discussion period closes tomorrow, we wanted to kindly follow up on our earlier responses. Based on your valuable suggestions, we have incorporated new experiments and analyses that we believe significantly strengthen the paper.

We would greatly appreciate it if you could let us know whether these updates address your remaining concerns.

AuthorsOpenReview ↗

Gentle Reminder - Deadline is Tonight

Dear Reviewer, We have addressed the concerns you raised.

We would greatly appreciate it if you could review our responses and consider increasing our scores as you see fit.

Thanks, The Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response: New Factor Analyses Validating the General Creativity Factor (c-factor)

A common concern for all reviewers centered around the validity of the general creativity factor (c-factor). Specifically, whether it reflects an authentic creative capability or is merely an artifact of model parameter scale or general capability (g-factor). We conduct three new factor analyses that reinforce our original finding: LLMs exhibit a domain-general creative capability that spans diverse tasks.

Summary of findings

Reviewers & Question ResolvedData & MethodologyTop-Line Result
Reviewers: i7tq, baFJ, 4hXo Q: Does the c-factor survive after controlling for model size/scale?Residual PCA (Creativity) 42 models × 6 creative tasks (Performance residualized for model scale)✅ C-factor survives scaling. It still explains 75.6% of residual variance. The eigenvalue remains confidently above noise levels (2.56x higher).
Reviewer: i7tq Q: What are the PCA results on standard, non-creative capability benchmarks?Residual PCA (Standard) 34 models × 6 standard tasks (Performance residualized for model scale)✅ Baseline g-factor confirmed. A parallel g-factor emerges independently, explaining 72.7% of residual variance.
Reviewers: i7tq, 4hXo Q: Does the c-factor survive alongside general capabilities, or is it just g-factor?Joint Factor Analysis 34 models × 12 tasks (Constrained Schmid-Leiman & Unconstrained Varimax)✅ Clean, orthogonal separation. C-factor uniquely explains 29.2% to 36.1% of variance with near-zero cross-loadings on standard benchmarks. It is a co-dominant capability, not a subordinate byproduct.

—

Baseline Paper Analysis

In the paper, we evaluated 83 LLMs across 6 creativity domains (spanning 67 text benchmarks). Conducting PCA on the 6 x 6 domain correlation matrix reveals a dominant first principal component (c-factor) explaining > 81% of the variance (Cronbach’s alpha = 0.96). This indicates that performance across diverse creative domains is unidimensional and highly correlated i.e., knowing a model's performance in one creative domain reliably predicts its performance in others.

—

The Core Reviewer Concerns

  1. Model Scale Confounder (Reviewers i7tq, baFJ, 4hXo): Does c-factor survive after controlling for model scale-based capability?
  2. Capability Baseline (Reviewer i7tq): What does PCA yield on standard "non-creativity" capability benchmarks?
  3. General Capability Confounder (Reviewers i7tq, 4hXo): Does c-factor survive a joint factor analysis alongside standard capability benchmarks, or is it subsumed by g-factor?

—

Methodology of Additional Analyses

Isolating Model Scale (Residual PCA):

To remove model scale as a confounder, we regress each task's performance against log(total parameters) across open-weight models with documented parameter counts:

Performance = θ₀ + θ₁ log₁₀(total parameters) + residual

We extract the residuals (representing model performance unexplained by size alone) and perform PCA on the residual correlation matrix. We apply this separately to both the 6 creativity domains (N = 42 open-weight models) and 6 standard capability benchmarks (N = 34 complete-case models).

Joint Factor Analysis (Schmid-Leiman Bifactor Decomposition):

To test whether c-factor separates from general capabilities, we evaluate the joint 34 models x 12 tasks system (6 creative domains + 6 standard capability benchmarks: GPQA, HLE, SciCode, IFBench, tau-Bench, CritPT). Using Schmid-Leiman orthogonalization on the size-residualized matrix, we decompose variance into three mutually orthogonal components: a global general factor (g-factor) loading across all 12 tasks, a specific creativity factor (c-factor) loading strictly on creative tasks, and a specific benchmark factor loading strictly on standard benchmarks.

Unconstrained Factor Rotation (Varimax PCA):

Because Schmid-Leiman structurally assigns tasks to their theoretical domains, we also conduct a fully unconstrained PCA analysis. We extract two principal components from the 34 x 12 size-residualized matrix without imposing any domain constraints. We then apply Varimax rotation to test whether a clean separation between creativity and general capability naturally emerges from the data.

Creativity Domains (Isolating Model Scale)

After regressing out model parameter scale across open-weight models (N = 42), the first principal component still explains 75.6% of the residual variance (compared to 82.6% unadjusted). The corresponding eigenvalue is 4.537 (compared to 4.957 unadjusted), with Cronbach's alpha = 0.935 (previously Cronbach’s alpha = 0.96). In parallel analysis over 1,000 random matrices, the 95th percentile noise threshold is 1.775. The observed eigenvalue of 4.537 is 2.56x above the noise ceiling. Therefore, model scale alone does not explain away the c-factor. A latent, general creative capability reliably explains model performance across the 6 creative domains.


Standard Capability Benchmarks (Baseline g-factor)

We conducted the identical residual PCA on the 6 standard capability benchmarks (N = 34). The first principal component explains 72.7% of the residual variance (compared to 82.7% unadjusted), with an eigenvalue of 4.362 (compared to 4.959 unadjusted, Cronbach's alpha = 0.923). Parallel analysis yields a 95th percentile noise threshold of 1.863. Because 4.362 is well above noise, standard capability benchmarks independently form a dominant g-factor.


Joint Bifactor Decomposition (Schmid-Leiman)

Applying Schmid-Leiman orthogonalization to the joint size-residualized matrix (N = 34, 12 tasks) decomposes performance into three mutually orthogonal factors. The global factor (g) explains 41.7% of common variance. The specific factor for the standard benchmarks explains 29.0% of common variance. Lastly, the specific factor the creativity tasks (the c-factor) explains 29.2% of common variance.

Crucially, this 29.2% of variance is strictly orthogonal to model scale, the global g-factor, and benchmark specific capabilities like instruction following, coding, scientific reasoning etc. In the unadjusted matrix, the g-factor explained 69.5% of common variance, while c-factor and the specific factor for standard benchmarks explained 15.2% and 15.4%, respectively. Removing model scale effects reduces the dominance of the global g-factor (69.5% -> 41.7%) and nearly doubles the unique variance explained by the creativity factor (15.2% -> 29.2%).


Unconstrained Factor Rotation (Varimax PCA)

Finally, an unconstrained 2-component PCA with Varimax rotation on the residualized matrix confirms that this separation is not an artifact of constraints enforced in the Schmid-Leiman analysis. The rotated solution cleanly separates specific factors for creativity and standard benchmark tasks. The creativity component (c-factor) has high loadings on creative domains (0.70 to 0.93) and near-zero cross-loadings on standard benchmarks (e.g., 0.08 on IFBench and 0.09 on GPQA). Conversely, the standard benchmark specific component has high loadings on standard benchmarks (0.76 to 0.90) and near-zero cross-loadings on creative tasks (e.g., 0.01 on Humor and 0.01 on Creative Problem Solving). The two components explain 36.1% and 38.2% of total variance (48.6% and 51.4% of retained variance), respectively.


Conclusion

Therefore, we conclude that c-factor is not a subordinate byproduct. It stands as a co-dominant, equal-strength pillar of model capability.

AuthorsOpenReview ↗

Summary of all responses

We sincerely thank all reviewers (i7tq, baFJ, 4hXo) for their constructive feedback and engagement with our work. We appreciate that the reviewers highlighted our core contributions, including the value of unifying 67 fragmented benchmarks across six domains into a common evaluation harness, psychometric calibration with Judge Response Theory (JRT), an open-weight scoring model (AGC-Judge), and extensive independent human validation with 416 participants.

The reviewers raised important questions regarding the construct validity of the general creativity factor (c-factor), domain balance, measurement validity, and the practical utility of AGC-Judge. We have addressed every concern thoroughly. The new empirical analyses we conducted in response to your feedback have further validated and strengthened our core findings. Given that these new analyses directly resolve all concerns and further substantiate our findings, we respectfully ask the reviewers to consider raising their scores.

Below, we summarize our responses across two tables: (1) concerns resolved through new analyses, and (2) concerns resolved through clarifications and existing evidence.


Concerns Addressed with New Analyses

ConcernReviewerSummaryStatus
Construct Validity of the Creativity Factor (c-factor)All (i7tq, baFJ, 4hXo)We conducted new factor analyses showing that the c-factor clearly survives after removing model scale effects and emerges alongside standard capability benchmarks as a co-dominant, equal-strength pillar of model capability.✅
Domain Imbalance & Task Composition SensitivitybaFJWe performed a subsampling analysis showing that strictly balancing task counts across all domains does not change our single-factor conclusions or leaderboard model rankings.✅
Creativity Differences Among Models with Equal General Intelligence4hXoWe demonstrated that among models with matched general performance, AGC-Bench reveals meaningful differences in creativity that are independently confirmed by human evaluations.✅

Concerns Addressed via Clarifications and Existing Evidence

ConcernReviewerSummaryStatus
Relationship to Prior Work (Cao et al. / CreataSet)baFJCreataSet is already evaluated as one of our 67 benchmark datasets, and we will explicitly cite and discuss its distinct scope in our related work section.✅
Measurement Validity vs. JRT Calibration (Length & Style Biases)baFJWe clarified that while JRT calibrates judge leniency, construct validity is independently established through objective formula-based benchmarks, length controls, and stylistic paraphrase tests.✅
Overcoming Limitations of Isolated Source Benchmarks4hXoWe detailed our quality control audits, evaluator bug fixes, and JRT calibration steps that overcome the idiosyncratic errors and noise of isolated benchmarks.✅
Practical Contribution and Cost Savings of AGC-Judge4hXoWe showed that AGC-Judge cuts queries in half, slashes evaluation costs from commercial APIs to pennies on rented GPUs, reduces AI self-preference bias, and reliably preserves leaderboard model rankings.✅

We believe these additions and clarifications resolve all outstanding concerns, and we will incorporate the new analyses, and discussions into the final revision of the paper. We thank the reviewers again for their time and valuable guidance.

AuthorsOpenReview ↗

Gentle Follow-up: Revisions and New Experiments

Dear Reviewers and Area Chair,

As the discussion period closes tomorrow, we wanted to kindly follow up on our earlier responses. Based on your valuable suggestions, we have incorporated new experiments and analyses that we believe significantly strengthen the paper.

We would greatly appreciate it if you could let us know whether these updates address your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.