acceptodds
Under review as a conference paper at ICLR 2027

JUDGEGYM: VERIFIER-ANCHORED ENVIRONMENTS FOR ATTRIBUTING LLM JUDGE FAILURES

Abstract

A key strength of LLM judges is that they scale evaluation. A judge scores answers no human has time to check, and the same judge serves as the reward model a policy is trained against. However, we find that the labels used to train and test judges are a source of blind spots. Judges are graded on preference labels, and a preference label says which answer was better and nothing about why, so a judge that misses a flipped sign scores the same as one that favors the longer answer. We identify two failure modes of this practice. First, an asserted label cannot attribute a failure to a mechanism, so training judges harder raises aggregate scores without removing the flaw. Second, the oracle that grades answers is lenient by design, and its tolerance silently deletes the test cases that would expose scale errors. We introduce JUDGEGYM, which builds judge tests from benchmarks whose expert written (gold) answers come with the short program that computes them. JUDGEGYM corrupts a gold program in one known way, re-executes it, and keeps the pair when the benchmark’s matcher says the answer changed. Both right verdicts are therefore computed rather than asserted. Each kind of corruption tests one invariant, and a control that changes nothing must always be discarded. Five kinds of change over 7,095 FinQA programs give 24,875 verified pairs, about a quarter of them with the sign flipped, and the control is discarded all 7,095 times. The benchmark’s matcher treats 0.125 and 12.5 as the same percentage. That leniency hides 60% of the changes to a constant’s scale, and a strict matcher recovers 99%. We prove that this hidden share caps what any model trained against the matcher can gain, at any sampling budget. Since every right response is known, the pairs are an exact training reward as well as a test. We compare JUDGEGYM with nine judge and reward-model benchmarks, specify a one-GPU training recipe and a finance-to-tax-law transfer experiment, and release the engine, the pairs, the harness, and every number as a recomputable artifact. An executed label, not a larger judge, is therefore what makes a judge’s failure attributable.

Primary area:
datasets and benchmarks
Keywords:
LLM-as-a-judge, reward models, judge evaluation, mutation testing, executable verification, verifiable rewards, counterfactual test cases, reinforcement learning for judges, financial question answering, statutory reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_EMuSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asserts that existing learned judges suffer from a lack of clean labels and issues with graders. To solve this issue, the authors propose corrupting "correct" samples into "guaranteed-incorrect" ones by programmatically mutating them using a set of predetermined operands, thus essentially creating new labels.

Critical strengths

The paper correctly identifies that judges are often trained on limited/poor data and that grading steps often have flaws which ought to be removed.

Critical weaknesses

The paper does not support several of its key claims, for example:

  • "We propose a way to build verified evaluation environments for LLM judges automatically from the executable structure of existing benchmarks" -> The paper proposes corrupting assumed-correct ground truth human labels to augment existing data with synthetic negatives. This is a well-known technique applied to a small subdomain.
  • The introduction claims that the authors run pre-registered experiments but does not actually report running any.
  • The proof of the "learning cap" is trivial and not helpful in practice.

The paper does not cite the long history of error-injection/perturbation for training robust judges and other models e.g., [1,2,3].

Lastly, the paper is very wordy and hard to follow, containing many AI-isms that obscure the underlying meaning.

[1] Kryściński, W., McCann, B., Xiong, C., & Socher, R. (2020). Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9332–9346. doi:10.18653/v1/2020.emnlp-main.750 [2] Sellam, T., Das, D., & Parikh, A. P. (2020). BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7881–7892. doi:10.18653/v1/2020.acl-main.704 [3] McAleese, N., Pokorny, R. M., Cerón Uribe, J. F., Nitishinskaya, E., Trębacz, M., & Leike, J. (2024). LLM critics help catch LLM bugs. arXiv preprint arXiv:2407.00215.

Questions for authors

Only fundamental changes to this paper would affect my assessment. I recommend a significant overhaul and resubmission in the future. My most important suggestions the authors would be to run the proposed experiments to ensure the synthetic labels actually allow for the training of meaningfully better judges, and citing and discussing the relevant literature to more clearly establish the novelty of the proposed approach.

Reviewer_qtf5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces JudgeGym, a verifier-anchored framework for constructing evaluation and training environments for LLM judges. It mutates executable gold programs, re-executes them, and keeps only mutations whose outputs differ under the benchmark matcher, producing automatically verified judge pairs without human annotation. On FinQA, the authors construct 24,875 verified pairs from 7,095 programs and analyze different corruption types and the effect of matcher tolerance. The paper also proposes a training protocol and cross-domain transfer experiments for judge models.

Critical strengths
  1. The verifier-anchored construction is well motivated. Executing program mutations provides objective labels and allows failures to be attributed to specific corruption families, which is a useful alternative to conventional preference-based judge benchmarks.
  2. The analysis of oracle tolerance is valuable. It clearly demonstrates that the choice of evaluation oracle can substantially affect what judge benchmarks measure.
Critical weaknesses
  1. The main empirical claim is not yet sufficiently validated. The paper proposes JudgeGym as an environment for evaluating and training LLM judges, but it does not evaluate any actual judge model.
  2. The coverage is quite narrow. The benchmark is primarily based on executable numerical reasoning in FinQA. It remains unclear whether the proposed framework generalizes to broader judging tasks, such as open-ended generation and agent trajectories.
Questions for authors

How well does JudgeGym transfer beyond executable numerical/program-based tasks?

Reviewer_bwFzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

JudgeGym builds benchmarks for LLM judges by deliberately introducing known mistakes into correct calculation programs. It generates wrong answers in existing benchmarks and then asks whether a judge can accept the original answer and reject the corrupted one.

Critical strengths

Interesting paper that aims to make LLM judge/reward model evaluation benchmarks more challenging.

Critical weaknesses
  1. It proposes benchmark modification, but no models were trained/evaluated on the proposed benchmark.
  2. Building challenging evaluation benchmark for llm judge/reward models is quite common in the literature, as cited by the authors. Therefore, the contribution is quite incremental.
Questions for authors
  1. Justify how the contribution in the paper is significant and interesting for the ICLR community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.