acceptodds
Withdrawn from ICLR 2027

SepaRank: Measuring Intelligence Beyond Human Scale

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How can we measure intelligence beyond human capability? Human-authored benchmarks saturate, and above human capability, examiners may not know which tasks are both hard and verifiable. We argue that this difficulty is inherent to absolute-scale evaluation and propose a new paradigm based on relative measurement in which models generate public challenges that separate other systems. We develop a novel, fully endogenous protocol that naturally scales with agent capabilities, requires no human-authored questions, labels or judgements, and is also judge-free. We instantiate this framework across verifiable and open-ended, non-verifiable domains, illustrating how model-generated evaluation can continue to measure systems beyond the human frontier.

Primary area:
datasets and benchmarks
Keywords:
AI evaluation, benchmarks, psychometrics, relative measurement, adversarial rating systems, judge-free adjudication, self-improvement, language models, multi-agent games

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_w23kOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces "adversarial psychometrics" as a pairwise protocol (a game) of problem proponents and solvers. The score is the average of the proponent score (the variance of the predictions of the other players) and the correctness scores (1 - Brier Score of the own probability predictions when a solver). This comes with experiments with two variants of the same domain (code with verifier or open-ended) with 10 modern LLMs as players of the game.

Critical strengths

Evaluating beyond human scale is becoming an important and urgent problem, but at the same time difficult.

Critical weaknesses
  • "Beyond human scale". The paper talks about an absolute scale (abstract), but this is about relative (populational) evaluation scales. The paper uses the term "human scale" (title), but there's nothing about humans or anything that suggests that this goes beyond the human scale. It just ranks models, as many other games would do. A chess tournament could possibly give a similar ranking.
  • Focus on Spearman, the g-factor and intelligence, when the connection with the protocol is very weak. The introduction is too generic, and starts with Spearman. This has been covered elsewhere and the connection with the paper is purely motivational. There are several (old) papers and books connecting all this, also about the g-factor and intelligence, and the use of psychometric approaches in AI evaluation. Then, including the results from the original Spearman's paper is pointless. This can be found elsewhere and more space should have been devoted to explaining the key concepts and results of the paper. Section 2 can be removed completely or moved to an appendix.
  • Use and comparison with psychometric techniques is limited and biased. "Most AI benchmarks inherit this psychometric structure". This is false, most don't even extract latent factors with factor analysis, IRT, etc., and especially most of the references used in that paragraph do not use any psychometric techniques. This is simply misleading.
  • Construct validity: I don't find a clear rationale why this should measure "intelligence". The hypothesis that a good intelligence discriminator has to display an important "component of intelligence" is a bold statement. Also, this doesn't clarify either whether this is a side effect (sufficient condition) of intelligence (if something is intelligent it will test intelligence well) or a necessary condition (for something to be intelligent it has to be test) or both).
  • Questionable assumptions: One major methodological issue is that the paper seems to criticise that "the common issue is that the challenger possesses information unavailable to the solver". No, this is actually the natural approach for an examiner to be able to test beyond their capacity, by constructing inverse problems, such as the example of the factorisation into two primes.
  • Uncertainty as variance and psychometric discrimination: "shared uncertainty among players is indicative of an uninformative challenge and should yields [sic] no separation reward" - this is actually what IRT and many other psychometric techniques do, trying to determine difficulty and discrimination from a population. Several parametric models map the information of an item with the uncertainty of the population. The connection with all this is missing, with the only psychometric technique being 100 years old (rudimentary factor analysis). The motivation that "the proposer succeeds only by finding questions that genuinely separate the audience" and targeting for variance are the traditional elements of psychometrics and the quality of items. In adaptive testing, a good evaluator is actually the one that chooses the item that better discriminates an individual or a population.
  • Contribution: Making this iterative or adversarial is not new, and as the authors recognise, already explored in MathDuels.
  • Lack of theoretical or empirical support for the protocol: The paper "conjectures" that variance on the probabilities discourages uninformative challenges, but this is not supported empirically or theoretically anywhere in the paper. The weakest part of the paper is that there's no theoretical justification of any of the protocols and no connection with other adversarial situations in game theory and multi-agent theory. Not even ablation of why J has to be defined as exactly the average of P and (1-B), where these two quantities (even if normalised to 1) have different sources and evolution.
  • Homogeneous group of agents: The experiments are limited to a quite homogeneous group of agents (I wonder what the results would be with more diversity or even adversarial players).
  • Rudimentary psychometrics: The comparison with some benchmarks and their g factor is a bit rudimentary, especially having more powerful indices of LLM capability, based on more modern psychometric techniques, such as Epoch's ECI based on IRT.

*** Some other comments or typos that may not be critical weaknesses but I think may be useful for the authors and I couldn't find a place where to put them.

Section 4 "answers A and ???" not A?

Also false commitments are left as future work, and preferences are not included.

Questions for authors
  1. The variance on the probabilities depend on the solvers to be calibrated. If they are not, random questions one could get more variance that meaningful questions. Can you try to use some uncalibrated models and see how this affects the results?
  2. Preferences could lead to variance: For instance, just asking about "red" being your preferred colour could lead to variance, but that doesn't mean that the test is measuring any intelligence (at least on the solver side), just information about the population. This means that even if for one experiment the metric converges that doesn't mean that it has to converge to any meaningful construct restricted to intelligence. All this may be artifices of the LLM population being not sufficiently diverse to be exploitable in other ways (or the prompts being limited to make things converge). Can you try with preferences or prompt the models to have some preferences and personalities and exploit this to have different answers?
  3. Do all the results of Figure 6 come from Artificial Analysis? Why the choice of these benchmarks and what are they measuring? For instance, AA-Omniscience is not even measuring a cognitive capability but rather the propensity of not answering a question when not known.
  4. Can I get more details about the models, the calibration of the models, their range in probabilities, etc. Also, to maximise the score, one model can try to regress to the population in case of doubt, and become less than perfectly calibrated could be optimal in the game. All these things should be analysed before getting the numbers from an experiments without no backing in any theory of intelligence or computational complexity.
  5. Given the long prompts and the complex game, good models could just get better because they understand the dynamics of the game better, rather than just because any intrinsic "intelligence" of the model. Can we clarify this?
  6. I may even wonder whether Astra or some other powerful models, given the equations in the prompt, have found optimal solutions to this "game" that could be incorporated as a theoretical support for this paper. I'm saying this also because GPT-6 Astra starts very high in P, in comparison with the others, even in the first round. Also because GPT-6 Astra may be using mind modelling (theory-of-mind) of the other players when the pseudonym is disclosed. Can we ask Astra to predict what the other agents are going to predict? If it performs well then the metric is loaded strongly by mind modelling (theory of mind of the other players).
Reviewer_fKnJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SepaRank, a model-generated evaluation framework where LLMs act as both problem proposers and solvers. Proposers are rewarded for inducing disagreement among solvers, while solvers are scored against the proposers' committed answers. The resulting rankings show correlations with existing benchmarks.

Critical strengths
  1. The idea of using population-level disagreement for automated model evaluation is interesting.
  2. The proposed framework is simple, scalable in design, and does not require human-authored questions or semantic judges.
  3. The experiments cover both verifiable and open-ended domains, with useful analyses of model behavior.
Critical weaknesses
  1. Insufficient evidence for scalability. The experiments only evaluate a fixed set of ten contemporary models. This does not establish that SepaRank can continue generating discriminative challenges as model capabilities improve, despite scalability being a central claim of the paper.

  2. Unclear construct validity. Disagreement among models does not necessarily reflect meaningful differences in intelligence. It may also arise from calibration differences, systematic biases, or ambiguous questions. Correlations with existing benchmarks across only ten models provide limited validation of the proposed measurement.

  3. Concerns about judge-free evaluation. Solvers are scored against proposers' commitments rather than ground-truth answers. In the code experiments, 23.8% of valid proposals have commitments that disagree with actual execution results. This raises concerns about whether the resulting scores genuinely reflect problem-solving ability.

Questions for authors
  1. Scalability: Could you evaluate SepaRank across different generations of models, from earlier LLMs to current frontier models, and examine whether the generated challenges become progressively harder while retaining discriminative power? This would directly strengthen the central scalability claim.

  2. Ground-truth validation: In the code experiments, could you recompute solver scores using actual execution results rather than proposers' commitments, and compare the resulting rankings? This would help quantify the validity cost of judge-free evaluation.

  3. Construct validity: Could you investigate whether high-disagreement questions genuinely separate models by objective correctness, rather than merely reflecting differences in confidence or calibration? This would strengthen the interpretation of SepaRank as an intelligence measure.

Reviewer_HXjcOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SepaRank, an framework in measuring AI intelligence beyond the human scale. In the experiment, each model acts as both a question proposer and a solver, and is given a separate score for each role. Proposers get higher scores when their questions lead other models to make different predictions. Solvers earn higher scores when they more accurately predict the answers that proposers committed to. The framework combines these two scores to rank the models, without requiring human-written questions or humans or other models to judge the answers. The experiments cover verifiable code tasks and open-domain questions evaluated without semantic judgements. The authors test and rank several existing large language models, and find that the resulting rankings correlate well with some established measures of model capability.

Critical strengths

The experimental design is quite novel. While the idea of models acting as both proposers and solvers is largely shared with MathDuels, other aspects, such as letting models choose the domains of their questions and asking solvers to predict the proposers’ committed answers without semantic evaluation, are particularly interesting. These choices allow the protocol to continue operating without human involvement even if models surpass human capabilities.

The overall rankings from code tasks and open-ended questions are highly correlated. This is consistent with the paper’s motivation and supports the framework’s applicability across different domains.

Critical weaknesses

1.The connection between the scores and intelligence needs further validation. Solvers are asked to predict the proposer’s committed answer, which may differ from the correct answer. This happens in 23.8% of valid proposals in the code experiments.

2.The experiments involve a limited number of models and do not examine different combinations of participating models.

3.There are several grammatical errors and typos in the main text. For example, some text appears to be missing after “and” on line 208. Figure 1 is also somewhat difficult to read.

Questions for authors

How would the results change if the code tasks were scored against the actual execution outputs rather than the proposers’ committed answers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.