acceptodds
Under review as a conference paper at ICLR 2027

Contextual agent evaluation with orthogonal equilibrium learning

Abstract

Contextual agent evaluation seeks to identify the best agents for each context from offline, selectively observed relative feedback. Existing score-based methods, such as Bradley–Terry, impose a transitive preference ordering, which fails to reflect collective preferences when human judgements are heterogeneous. Inspired by social choice theory, we frame evaluation as a contextual game between two players, each selecting a distribution over agents as the strategy to receive greater collective preference than the other. Then, the support of the Nash equilibrium defines a context-specific set of winners. However, learning context-specific equilibria from offline logs is difficult because each context reveals human feedback on only a subset of agents, and na\"ive plug-in estimators can therefore be biased. To address these challenges, we propose NashEval, a general framework for robust contextual equilibrium learning. NashEval first constructs debiased estimates of the contextual payoff matrix that characterizes the game. It then learns the context-to-equilibrium mapping with a tailored orthogonal loss, which avoids the need to solve a separate game for each context. We show theoretically that errors in estimating the nuisance functions underlying the payoff matrix affect the regret of the learned equilibrium (i.e., exploitability) only through higher-order terms. Empirically, NashEval improves robustness of equilibrium learning and adaptively identifies the set of top-performing agents across contexts. In summary, our framework enables context-aware agent evaluation that reflects diverse preferences across the population.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
agent evaluation, learning from relative feedback, intransitive preferences, zero-sum games, debiased machine learning, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Y2HnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studies how to perform such context-dependent agent evaluation from offline feedback. This work propose NASHEVAL, a general framework for robust contextual equilibrium learning. NASHEVAL first constructs debiased estimates of the contextual payoff matrix that characterizes the game. NASHEVAL then learns the context-to-equilibrium mapping with a tailored orthogonal loss, which avoids the need to solve a separate game for each context.

Critical strengths

(1) This work propose NASHEVAL, a novel game-theoretic framework for learning context-dependent agent evaluators from logged data with relative feedback.

(2) This work derive strong theoretical guarantees for our equilibrium learner in the form of Neyman-orthogonality, bounds on the average exploitability, and a quasi-oracle convergence rate. (3) This work demonstrate the effectiveness of our NASHEVAL framework on both simulated and real-world datasets.

Critical weaknesses
  1. Experimental evidence should better support real-world claims.
  2. Result analysis and limitations should be more precise.
Questions for authors
  1. The paper’s core contribution is a contextual, debiased equilibrium learner, but the novelty relative to existing maximal-lottery methods (Lanctot et al., 2023; Khalaf et al., 2026) and contextual BT methods (Frick et al., 2025; Ong et al., 2025) needs sharper formal positioning. In particular, please add a precise comparison table or proposition clarifying which assumptions each method makes: transitivity vs intransitivity, global vs contextual, offline selective feedback vs interactive queries, and plug-in vs orthogonal nuisance estimation. It would also strengthen the paper to include a controlled experiment isolating the benefit of the DPM from the benefit of the orthogonal gap loss.
  2. The empirical results limit claims about “real-world” contextual evaluation. Please conduct a sensitivity analysis to judge heterogeneity and menu coverage/positivity.
1 comment

Dear Reviewer,

We sincerely thank you for reviewing our paper. We hope that our response and the revised manuscript have addressed your concerns. If so, we would be grateful if you could consider updating your score. If otherwise, please let us know, and we would be happy to clarify within the remaining discussion period.

Thank you again for your time and effort!

The authors

Reviewer_PzGEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes NashEval, a framework for context-dependent agent evaluation from incomplete preference feedback. It accommodates cyclic collective preferences through regularized zero-sum games and learns a context-to-equilibrium mapping using debiased payoff estimates and an orthogonal gap loss. The paper establishes exploitability and convergence guarantees under nuisance-estimation assumptions and evaluates the method on synthetic data, LLMs, and reinforcement learning agents.

Critical strengths
  • The motivation is compelling: collective preferences can be cyclic, making game equilibria a natural choice for agent evaluation.
  • The theoretical analysis and experiments are comprehensive, making this a solid and well-developed study.
Critical weaknesses
  • The technical novelty is somewhat limited. The debiasing term follows the standard doubly robust construction, and Theorem 3 mainly combines existing tools.
  • The guarantees are modular: accurate preference, propensity, and payoff estimation imply accurate equilibrium learning. However, the required estimation rates are assumed rather than established under concrete structural conditions.
Questions for authors

None

2 comments

Dear Reviewer,

We sincerely thank you for reviewing our paper. We hope that our response and the revised manuscript have addressed your concerns. If so, we would be grateful if you could consider updating your score. If otherwise, please let us know, and we would be happy to clarify within the remaining discussion period.

Thank you again for your time and effort!

The authors

Thanks for your reply. I will maintain my positive score.

Reviewer_gUeMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies context-dependent agent evaluation from offline relative feedback (pairwise preferences, partial rankings, winner-of-subset) that is only observed on the menus of agents shown in each context. Since aggregating heterogeneous preferences can produce intransitive collective preferences, the authors formulate evaluation as a regularized symmetric two-player zero-sum game: for each context xx, a contextual payoff matrix A(x)A(x) aggregates mean pairwise outcomes across menus, and the support of the unique regularized equilibrium π∗(x)\pi^*(x) defines the set of winning agents.

The proposed method, NashEval, has two stages:

  • Debiased payoff matrix. An efficient influence function based pseudo-outcome is regressed on the context. The resulting payoff matrix is shown to be Neyman-orthogonal with respect to the outcome and selection nuisances (Theorem 1).

  • Amortized equilibrium learning. A context-to-equilibrium network is trained by minimizing the regularized gap function under this payoff matrix, which avoids solving a separate game for each context.

The theory establishes universal Neyman-orthogonality of the gap loss (Theorem 2) and an exploitability bound in which nuisance errors enter only at higher order (Theorem 3). Experiments on a synthetic dataset, RouterBench (with preferences constructed from benchmark scores and LLM judges), a multi-objective highway-driving benchmark (with three linear judges), and descriptively on LMArena show that NashEval-debiased attains the lowest exploitability among the compared methods.

Critical strengths
  • Representing feedback as ordered partitions lets a single estimator handle pairwise preferences, full and weak rankings, top-L choices, and winner-of-subset feedback, and combine them in one payoff matrix. This matters in practice because real evaluation logs often mix modalities (such as arena-style duels alongside ranked or best-of-n judgments), and most existing evaluators handle only pairwise data. The strongly convex regularizer further makes the target π∗(x)\pi^*(x) a well-defined, single-valued function of the context, so learning it is a well-posed problem. Its Lipschitz stability (Prop. 1) also ensures that payoff estimation error translates into bounded equilibrium error. As Appendix C.5.1 shows, without regularization an arbitrarily small perturbation of the payoff can flip the selected winner, which would make any equilibrium learned from noisy, partially observed feedback unreliable. These properties underpin all of the paper's theoretical results.

  • Learning the context-to-equilibrium mapping through a differentiable gap loss has two consequences. Practically, it produces a policy that returns an evaluation for a new context with a single forward pass, which suits applications such as model routing that require per-prompt decisions at scale and low latency. Methodologically, it recasts equilibrium computation as risk minimization, which is what places the problem within the orthogonal statistical learning framework. The orthogonality, exploitability, and quasi-oracle results (Theorems 2–3, Corollary 1) are stated for this loss and would not apply directly to a pipeline that estimates the payoff matrix and then solves each game separately.

Critical weaknesses
  1. The intransitivity motivation is not validated on human data. The main motivation is that heterogeneous human preferences are intransitive, so Bradley-Terry style models are misspecified. However:
  • In RouterBench, preferences are constructed from benchmark correctness and three LLM judges assigned different emphases, a design intended to induce intransitivity. The judge models are not identified.
  • In MO-Highway, preferences come from three hand-designed linear utilities, which produce cycles in 11 of 32 test configurations.
  • The only human data, LMArena, has no ground truth and is used descriptively. The paper does not report how frequently cycles occur in it.

The experiments therefore show that NashEval recovers constructed games well, not that intransitivity is prevalent or consequential in real human feedback.

  1. The paper describes the predicted winner frequencies as largely consistent with the reference. In the Chat/IF column, however, the predicted frequencies sum to about 302% versus about 152% for the reference, meaning the predicted supports are roughly twice as large. Gemini 2.5 Pro is predicted to be a winner in 79% of Chat/IF prompts versus 21% in the reference, and Qwen3 235B in 35% versus 13%. In the math column, Gemini 2.5 Pro is predicted at 56% versus 30%. Agreement is reasonable for Knowledge and Programming but not for Chat/IF, which makes the consistency claimed for Figure 4 overstated.
Questions for authors
  1. Can the authors add a synthetic experiment in which selection depends strongly on the context, where the debiased payoff matrix should help most? How does the gap between the debiased and plug-in estimators vary with the strength of this dependence?

  2. Which LLMs serve as judges in RouterBench, and how sensitive are the results to the judge emphases?

  3. How does the amortized policy compare in runtime and accuracy with an efficient solver for the regularized game, such as a QP solver or mirror descent, rather than support enumeration?

1 comment

Dear Reviewer,

We sincerely thank you for reviewing our paper. We hope that our response and the revised manuscript have addressed your concerns. If so, we would be grateful if you could consider updating your score. If otherwise, please let us know, and we would be happy to clarify within the remaining discussion period.

Thank you again for your time and effort!

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.