acceptodds
Under review as a conference paper at ICLR 2027

Codifying the Judge: Scalable Evaluation via Program Distillation

Abstract

LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, inference latency, and opaque decisions—limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at evaluation time, we distill its decision logic into a committee of programs that can score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and eight model families, we show that programmatic judges match the performance of a 13B-size LLM judge at 47 higher throughput. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Evaluation, Program Distillation, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TPKFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PAJAMA, a method that generate simple Python scoring functions from evaluation rules and labeled examples. It combines their scores by a weak-supervision model and uses an LLM to judge difficult cases when the results are uncertain. The authors evaluated PAJAMA on five preference datasets, and compared it with different LLMs, and use its labels to train reward models. They also study how changes in the dataset affect bias. The standalone committee achieves 78.11% accuracy and processes 439.41 comparisons per second. This hybrid system offers a better balance between accuracy and speed. And the reward models that trained on labels from these Python programs also perform better on RewardBench than models trained on GPT-4 labels.

Critical strengths

1.It reframes repeated preference evaluation as an execution of reusable programs, it also addressed the cost of applying an LLM judge to each response pair. It combines those programs with the selective LLM fallback, making the efficiency gain useful even when the programs cannot handle every comparison reliably.

2.It makes its scoring rules as editable code, making the behavior of the evaluator easier to follow and revise. It also cost lesser when evaluating the reward-model training, showing that the generated signals are valuable beyond agreement on a preference benchmark.

Critical weaknesses

1.It hasn’t confirmed whether its program can get the substance right independently of surface cues related to preference labels. 2.Under similar supervision, PAJAMA's contribution to synthetic judgment rules still yet to be resolved compared to simpler heuristic evaluators.

Questions for authors

1.What proof is there that PAJAMA can tell if something’s actually correct without relying on how it looks or is formatted?
2.Why is program synthesis better than just using simple rule-based judges when you have the same amount of resources?
3.How well does the routing system catch program mistakes when you switch up the type of problems it’s evaluating?

4 comments
AuthorsOpenReview ↗

Reviewer TPKF (1/2)

Thank you for your positive review! We are glad you found the paper clear and easy to follow, as the other reviewers also noted. We appreciate your recognition of the key strengths of our work: low cost, high throughput evaluation. When combined with LLM judges, PAJAMA uses program outputs as routing signals to escalate uncertain samples, with no additional inference latency. Programs also make evaluation logic explicit and traceable, so current coding agents can calibrate them easily. Finally, reward models trained on program labels outperform those trained on proprietary-model labels, at 45–50× lower cost.

  • On Direct Evidence for Programmatic Judges.
    • First, since the synthesized judges are programs, we can read their actual decision path directly from the code. For any verdict, we can point users to the exact lines that produced it. This also lets us calibrate or remove programs that rely heavily on surface cues. To test this, we removed every program synthesized from the three rubrics most closely tied to surface cues: (i) factual accuracy indicators, (ii) formatting, and (iii) evidence density. We then re-ran our experiments and report the results below. Average test accuracy drops by only 1.0 point. Accuracy is unchanged on JudgeLM and even improves on MultiPref (+1.0) and Preference-700K (+1.0).
    • We also note that model-based evaluators such as LLM judges are harder to audit than PAJAMA. Likewise, there is no direct proof that LLM judges reach correct verdicts without relying on surface cues. They are also sensitive to stylistic biases, and their stated justifications may not reflect their actual decision process, as discussed in our introduction. Programs, in contrast, can be audited and edited directly. As Section 4.4 shows, coding agents can further calibrate them to improve robustness while maintaining performance. We report the extended calibration results in our response to Reviewer S8T2.
    • Below we show a concrete example that we can provide to users. We use the best-performing program on Prometheus (judge_41, 324 lines in total), which computes 12 weighted features targeting clarity and tone written by Opus 4.6.

Given Sample

Query: You are a school counselor… "I've been feeling really down lately. I think my friends don't like me anymore… I feel so left out and lonely."

Response 1 (human score: 1/5): "Well, life is tough. Maybe you're just too sensitive. It's important to just get over it and move on. After all, there are many people in the world. Maybe you just need to find better friends."

Response 2 (human score: 4/5): "I can see that you're going through a tough time and feeling left out. It's important that you share these feelings with a trusted adult or a professional counselor."

The program scores Response 1 at 3.36 and Response 2 at 8.11, matching the human preference. Notably, the preferred response is shorter, so a length heuristic would choose wrongly. This shows that synthesized programs capture judging logic beyond surface cues such as length. The decision is driven by interpretable signals:

FeatureResponse 1Response 2Contribution to the gap
Filler-word density (well, maybe, just)0.001.00+1.50
Dismissive phrases ("get over it", "move on")0.401.00+0.60
Emotion-aware penalty (active only when the query expresses distress)−0.200.00+0.20
Empathy markers ("I can see")0.400.60+0.14
Final score3.368.11

Removing the flagged rubrics costs little accuracy (test accuracy, %).

Program poolJudgeLMPandaLMMultiPrefPrometheusPreference-700KAvg.
All rubrics81.170.472.288.867.876.1
w/o flagged rubrics81.168.873.283.468.875.1
AuthorsOpenReview ↗

Reviewer TPKF (2/2)

  • On Simpler Heuristics.
    • In our preliminary work, we explored this idea extensively: we brainstormed many heuristic rules and evaluated them on the preference datasets. As a concrete example, we compare a length-based heuristic against the best single program synthesized by Claude Opus 4.6. The best program encodes far more complex decision logic (e.g., bigram/trigram redundancy, root type-token ratio, and sentiment markers) and handles edge cases that humans cannot easily write down. Writing such rules is cognitively demanding and time-consuming even for human experts, since it requires a deep understanding of each dataset. Most importantly, each condition must be tuned iteratively by hand to get it right. Even so, a single program cannot reliably cover all samples. This is why PAJAMA draws on collective expertise: it distills the LLM's judgment into a diverse committee of programs across evaluation rubrics and aggregates their verdicts into a final decision.
    • We also measured the size of the selected programs. As shown in the table below, each program contains about 295 lines of code on average (up to 410), and each dataset's committee totals 2.4K–6.3K lines of evaluation logic. Writing and maintaining rules at this scale by hand for every new dataset would be impractical, whereas PAJAMA synthesizes them automatically.

Simpler heuristics vs. synthesized programs (test accuracy, %; coverage in parentheses).

Dataset"Longer response wins"Best single programPAJAMA
JudgeLM49.589.2 (50% cov.)81.1 (99% cov.)
PandaLM67.273.7 (65% cov.)70.4 (89% cov.)
MultiPref74.079.3 (50% cov.)72.2 (96% cov.)
Prometheus65.093.5 (55% cov.)88.8 (96% cov.)
Preference-700K66.773.3 (57% cov.)67.8 (96% cov.)
Avg. Perf.64.581.8 (55% cov.)76.1 (95% cov.)

Synthesized programs are large: about 295 lines each on average.

DatasetSelected programsLines per program, mean (min–max)Committee total lines
JudgeLM21300 (233–373)6,297
PandaLM14265 (161–337)3,709
MultiPref16309 (242–410)4,939
Prometheus8305 (251–344)2,438
Preference-700K15294 (223–359)4,411
  • On Routing Under a Change of Problem Type.
    • We thank the reviewer for this question. We answer this through two experiments, using all 20 cross-dataset pairs (programs built for one dataset, applied to another).
      • (i) Programs transfer. In Appendix E.5, we apply each dataset's programs to every other dataset without re-synthesis. Programs lose only 1.6 accuracy points on average; 15 of 20 pairs stay within 2 points of in-domain programs, and 4 pairs even match or exceed them. Synthesized programs capture largely reusable judging criteria: a program library built once can serve new datasets with only light recalibration.
      • (ii) The router still catches program mistakes. In this new experiment, we send the 20% least-confident samples to the LLM judge and count how many of the committee's errors they contain. Random routing would catch 20%. We test two cases:
        • Light refit (re-running only thresholds, selection, and aggregator on the new data; ~15 s): the router catches 32.8% of errors, close to the in-domain 35.7%.
        • No refit (the whole pipeline applied unchanged): the router still catches 27.1%, still well above random.
      • Overall, we show that our synthesized programs generalize well to new datasets and problem types, and with a quick recalibration on the new data, our router remains nearly as effective as in-domain.

Synthesized programs transfer across datasets.

Programs from ↓ / Evaluated on →JudgeLMPandaLMMultiPrefPrometheusPreference-700KAvg. change
JudgeLM82.170.1 (−2.0)72.3 (−0.8)89.6 (−1.4)68.1 (−0.7)−1.2
PandaLM82.3 (+0.2)72.171.0 (−2.1)89.2 (−1.8)69.0 (+0.2)−0.9
MultiPref80.4 (−1.7)71.4 (−0.7)73.186.9 (−4.1)68.9 (+0.1)−1.6
Prometheus80.9 (−1.2)70.9 (−1.3)73.5 (+0.4)91.068.4 (−0.4)−0.6
Preference-700K75.2 (−6.9)71.0 (−1.1)71.3 (−1.8)85.8 (−5.2)68.8−3.8

The router still catches program mistakes on new datasets.

RouterData% of committee errors caught ↑vs. random
Random—20.0%1.00×
PAJAMASame dataset (in-domain)35.7%1.79×
PAJAMANew dataset, light refit32.8%1.64×
PAJAMANew dataset, no refit27.1%1.36×

Dear Reviewer TPKF,

To further address your question about simpler heuristics, we ran a new experiment. We train logistic regression (LR) on the same validation split that PAJAMA uses, with two feature sets: 8 handcrafted text features and unigram TF-IDF features.

  • Handcrafted text features. We use the 8 heuristic listed below. Each feature is computed for both responses, and the classifier receives the difference between the two scores, so it learns which response has more or less of each feature. This setup gives a direct comparison between simpler features and features distilled into programs.
  • TF-IDF. We also use unigram TF-IDF features, trained the same way.

For each setting, we train LR over 5 trials and report the mean and standard deviation of test accuracy.

#FeatureWhat it measuresRationale
1charsNumber of charactersLength (longer answers are often preferred)
2wordsNumber of wordsLength; together with chars, also captures word length
3sentencesNumber of sentencesAmount of content
4avg_sent_lenWords per sentenceWriting style (dense vs. terse)
5bulletsNumber of bullet lines (-, *, •)Formatting
6headersNumber of markdown headers (#)Formatting
7digitsNumber of digit charactersSpecificity (numbers, dates, steps)
8urls_citationsNumber of URLs and citation markers (e.g., [1], "et al.")Apparent credibility

New results (test accuracy, %)

DatasetText Features + LRTF-IDF (1-gram) + LRPAJAMA
JudgeLM61.3 ± 0.578.8 ± 0.481.1
PandaLM51.4 ± 7.856.1 ± 0.470.4
MultiPref63.8 ± 1.065.7 ± 1.272.2
Prometheus75.0 ± 0.772.2 ± 0.288.8
Preference-700K66.4 ± 0.263.9 ± 0.667.8
Average63.6 ± 1.667.3 ± 0.376.1

PAJAMA achieves the best accuracy on every dataset and the best average. On average, it outperforms TF-IDF + LR by 8.8 points and handcrafted features + LR by 12.5 points. These results show the value of program distillation: it draws on the reasoning and coding abilities of current LLMs to produce richer evaluation signals than either hand-designed heuristics or bag-of-words features. The heuristic features mainly capture surface-level properties such as length and formatting, and TF-IDF captures only lexical cues. In contrast, PAJAMA's programs encode the judging logic of LLM evaluators, which lets PAJAMA go beyond these surface signals and generalize better.


We also have a new experiment that show how programmatic judges apply on factuality datasets. We use the Factuality subset of RewardBench-2, where one response in each pair contains a factual error. We keep only pairs whose two responses are matched in length, so that length cannot serve as a shortcut. We measure how many pairs the program committee judges on its own, how accurate it is on those pairs, and how well the resulting hybrid performs. The program committee evaluates each pair and routes the pairs it is least confident about to an LLM judge. We pair it with three LLM judges: OLMo-2-13B-Instruct, Qwen2.5-14B-Instruct, and Gemma-3-12B-Instruct.

SettingProgram correct (%)Program wrong (%)OLMo-2-13B (Acc. / Thr.)Qwen2.5-14B (Acc. / Thr.)Gemma-3-12B (Acc. / Thr.)
LLM only (route 100%)——52.0 / 4.871.4 / 4.457.7 / 4.7
Route 10%67.332.766.1 (+14.1) / 22.3 (4.6×)69.0 (−2.4) / 20.4 (4.7×)65.3 (+7.6) / 21.8 (4.6×)
Route 20%71.628.467.7 (+15.7) / 14.3 (3.0×)72.2 (+0.8) / 13.0 (3.0×)64.9 (+7.2) / 13.9 (3.0×)
Route 30%76.024.064.9 (+12.9) / 9.4 (1.9×)72.2 (+0.8) / 8.5 (1.9×)63.3 (+5.6) / 9.1 (1.9×)
Route 40%78.221.859.7 (+7.7) / 6.8 (1.4×)71.4 (0.0) / 6.2 (1.4×)58.9 (+1.2) / 6.7 (1.4×)

Program correct / wrong: accuracy and error rate on the pairs the committee labels itself. Acc.: hybrid accuracy (%); Thr.: throughput (samples/sec).

On length-matched factuality pairs, the hybrid with 20% routing outperforms each of the three 12–14B LLM judges used alone, at about 3× their throughput. This shows that programs complement LLM judges, and that they assess samples by more than surface cues or formatting features. As more pairs are routed, the committee’s error rate on the pairs it keeps drops steadily, from 32.7% to 21.8%. This suggests that its confidence scores reliably identify the cases it cannot handle and hand them off to the LLM judge.

Dear Reviewer TPKF,

As the discussion period is drawing to a close, we wanted to follow up on your questions. In addition to our initial response, we have run two new experiments to address your concerns.

These results show: (i) how decision paths as a evidence can be presented to users; (ii) the direct benefit of program distillation over simpler heuristics; (iii) improved performance on factuality and reasoning datasets; and (iv) how the routing system generalizes to different types of questions. We would be glad to hear your thoughts and are happy to answer any further questions.

Thank you for your time and for your valuable feedback.

Reviewer_S8T2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper targets the cost, latency, opacity, and bias of LLM-as-a-judge by proposing PAJAMA, which distills an LLM's judging logic into a committee of Python programs rather than calling the LLM on every sample. Ten evaluation rubrics are rotated into a synthesis prompt together with few-shot examples to generate functions of the form judging_function(query, response). Program scores are normalized, converted into +1/-1/abstain votes through per-program abstention thresholds, pruned by discarding programs below 50% validation accuracy, and the top-k survivors are aggregated with a Snorkel-style label model. A fallback mechanism routes only samples whose vote variance or aggregator posterior signals uncertainty to an LLM judge. Across five preference datasets and nineteen LLM judges, the program committee reaches 78.11% average accuracy at 439 samples/sec, matching OLMo-2-13B at 47 times the throughput, and the hybrid variant sits above the LLM-only Pareto frontier at every throughput level. A reward model trained on program labels outperforms one trained on GPT-4 labels on RewardBench by +3.6 and +4.05 points at 45–50 times lower labeling cost. Across five bias types the committee has the lowest average flip rate, and editing the programs with Claude Code lowers it further.

Critical strengths
  • The problem framing is interesting and the remedy is simple. The O(1) v.s. O(n) cost argument is persuasive.
  • The throughput advantage is overwhelming.
  • The routing signal is free. Aggregator posterior and vote variance, derived from program outputs with no additional model inference.
Critical weaknesses
  • The reward model cost comparison uses GPT-4 from 2023 and this is an unfair baseline in 2026. For example, GLM-5 is roughly 20x cheaper than GPT-4 and offers much higher intelligence. Pitting 2023 GPT-4 labels against programs synthesized by 2026 Opus 4.6 also confounds the method effect with the model effect; since the synthesizer ablation shows that changing the synthesizer alone moves accuracy by 8.3 points, it is impossible to tell which factor drives the +3.6/+4.05 RewardBench gains.
  • The "lowest average flip rate" headline in the bias study is an artifact of position bias. Because programs score each response independently, a 0% position flip rate follows trivially from the design, yet it is included in the average. Excluding position, PAJAMA’s average flip rate over the four content biases is about 15.1%, worse than Gemma-3-12B (about 6.1%), Gemma-3-4B (about 7.3%), and Qwen2.5-14B (about 9.5%). The LLM judges usually have response order randomized, which neutralizes position bias in practice.
  • Because the programs are surface heuristics rather than semantic judge, they are biased and easily exploitable when used as a reward. Rubrics 4, 9, and 10 explicitly instruct the synthesizer to reward citations, numbers, named entities, numbered lists, and headers. However, these are exactly the reference and rich-content biases, and the calibration prompt instructs not to reward citations: directly contradicts rubric 4. Programs restricted to standard-library string operations can be defeated by an RL policy that simply appends such patterns, and the LLMBar adversarial subsets of RewardBench already hint at this, with the Prometheus-trained PAJAMA reward model scoring 14.93% and 16.30% on two of them, far below the 50% random baseline.
Questions for authors

Adding a baseline that labels the 20,000 pairs directly with a low-cost contemporary model such as GLM-5.2. What would the cost and scores be?

4 comments
AuthorsOpenReview ↗

Reviewer S8T2 (1/2)

We thank the reviewer for the thoughtful feedback and for finding the problem interesting and our solution simple. We appreciate the recognition of our core advantage: PAJAMA replaces repeated model inference with one-time program synthesis and local execution, which yields large gains in throughput and API cost. Our routing mechanism is also practical. It reuses program outputs as routing signals, so the hybrid evaluation adds no inference cost for routing.

  • On Reward Model Baselines.
    • Thank you for the suggestion! First, we clarify that the GPT-4 labels we use are the ones released by the original JudgeLM and Prometheus papers (see Section 4.3 for our setup). Those labels were generated with carefully curated rubrics and show strong human alignment: JudgeLM reports over 90% agreement with human judgments, and Prometheus reports a Pearson correlation of 0.897. Both papers also used these labels to fine-tune strong LLM judges, which shows the labels are high quality. We consider them a strong, competitive baseline.
    • Second, during the rebuttal period we added an experiment with a more recent proprietary model, GPT-5.5, as the labeling source, and trained reward models on its labels. As shown below, GPT-5.5 labels improve over the original GPT-4 labels on both datasets, but PAJAMA still outperforms them on average at about 15× lower cost. Most importantly, the cost advantage grows with dataset size, because synthesis is a fixed one-time cost. This is the O(1) vs. O(n) benefit you highlighted.
Labeling SourceAPI CallsChatChat HardReasoningSafetyAverage
Trained on Prometheus Dataset
GPT-4 (provided labels)20,000 samples68.4438.8255.7045.9552.23
GPT-5.5 (new)20,000 samples78.7736.1856.1048.3254.84
PAJAMA80 programs79.3330.0460.5853.3855.83
Trained on JudgeLM Dataset
GPT-4 (provided labels)20,000 samples67.8838.8265.1358.2457.52
GPT-5.5 (new)20,000 samples68.9938.3868.5659.6358.89
PAJAMA80 programs67.8844.5272.9160.9561.57
  • On Position Bias and Content Biases.
    • First, position bias is an inherent weakness of LLM judges: their verdict can change when the two responses are swapped. In our experiments, we ran three trials with the response order randomly swapped, and the flip rate remained high. Programmatic judges avoid this bias by design.
    • Second, we extended our program calibration experiment by two more rounds targeting the remaining content biases. In each round, Claude Opus 4.6 audits and edits the programs from the previous round. Results are shown below. Each additional round reduces average bias further: average FR drops from 15.12 to 4.11, and average BWR drops from 39.78 to 33.96. After these rounds, program judges have a lower flip rate than Qwen2.5-14B-Instruct and Gemma-3-12B-Instruct. These averages exclude position bias. The edited programs also keep comparable evaluation accuracy, which changes from 88.8 to 87.2 on the Prometheus test set. These highlights advantage of programs: their evaluation logic is explicit and auditable, so biased features can be found and fixed directly. In contrast, for any model-based evaluator, it is hard to inspect the weights, locate a biased feature, and correct it directly.
MethodRich Content FR / BWR ↓Reference FR / BWR ↓Gender FR / BWR ↓Verbosity FR / BWR ↓Average FR / BWR ↓Test Acc. ↑Coverage
Round 0 (uncalibrated, in paper)23.13 / 48.9220.30 / 47.107.69 / 46.279.34 / 16.8415.12 / 39.7888.895.9
Round 1 (in paper)13.74 / 53.2817.29 / 43.173.70 / 36.173.25 / 9.689.50 / 35.5887.595.8
Round 2 (new)10.37 / 47.792.96 / 44.932.99 / 37.962.23 / 8.204.64 / 34.7287.195.9
Round 3 (new)6.77 / 44.032.96 / 44.204.48 / 39.422.23 / 8.204.11 / 33.9687.295.7
AuthorsOpenReview ↗

Reviewer S8T2 (2/2)

  • On Surface Heuristics.
    • First, we clarify that the rubrics define only the candidate pool, not the final judge. Each rubric yields 8 candidate programs, for 80 per dataset. We then normalize program scores, tune a per-program abstention threshold, and keep only programs that agree with held-out labels. Only 74 of the 400 candidates (18.5%) survive. Selection is therefore driven by validation data rather than by rubric text: for example, only 6 of the 40 formatting-rubric programs are kept.
    • Second, to test this directly, we removed every program derived from the three flagged rubrics (H4 factual accuracy indicators, H9 formatting, H10 evidence density) and re-ran selection on the remaining pool. Average test accuracy drops by only 1.0 point. Accuracy is unchanged on JudgeLM and improves on MultiPref (+1.0) and Preference-700K (+1.0). The largest drop is on Prometheus, whose grading rubric explicitly rewards specific evidence. Finally, when surviving programs do reward specificity markers, those biases can be corrected because the programs are human-readable and auditable (see Section 4.4 and the calibration rounds above). LLM judges do not offer such advantage.
Program poolJudgeLMPandaLMMultiPrefPrometheusPreference-700KAvg.
All rubrics81.170.472.288.867.876.1
w/o flagged rubrics81.168.873.283.468.875.1

Thank you for sharing interesting new results. I was actually curious about the cost and performance of using a newer model that is both cheaper and stronger than GPT-5.5 as the LLM judge (as you know, this field is advancing so quickly that I believe there are many better options than gpt-5.5). In any case, PAJAMA's cost advantage at larger data scales still holds, so I will not press this point further. It is also a nice observation that additional revision rounds improve the flip rate without significantly affecting test accuracy. I will raise my score to 3.

Dear Reviewer S8T2,

Thank you for your time and support. We are also excited to see our new results on the reward model distilled from GPT-5.5 and further program calibration, which do not harm evaluation performance!

The core idea PAJAMA wants to bring is that it shifts repeated model-based inference calls into a one-time synthesized program sets that can be executed locally (unlimited for any future run). We also want to highlight that evaluation is not just about accuracy. Latency, throughput, cost, and whether we can trace the underlying decision path also matter for performance. As our evaluation moves online and becomes more iterative, the drawbacks of LLM judges are hard to ignore.

We appreciate your reviews, which help us improve our paper. Please let us know if you have any further questions. We are happy to address them.

Reviewer_3dPaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAJAMA synthesizes lightweight Python judges, calibrates and aggregates their votes, and routes uncertain samples to LLMs. Experiments across five preference datasets assess accuracy and throughput, reward model supervision, and bias robustness, showing substantial efficiency gains with competitive accuracy.

Critical strengths

PAJAMA achieves 78.11% accuracy versus OLMo-2-13B’s 79.70% at 47× throughput.

Combining programs with OLMo-2-7B improves accuracy by approximately five percentage points at 2.9× throughput.

Reward models trained on PAJAMA labels outperform those trained on GPT-4 labels by 3.60–4.05 points on RewardBench, at 45–50× lower estimated labeling API cost.

Critical weaknesses

The paper does not compare PAJAMA with a simple classifier using similar text features and the same amount of labeled data. This makes it unclear whether generating programs provides additional benefits.

The paper does not show whether the programs can detect factual or reasoning errors when answers have similar length and formatting. Their ability to judge correctness therefore remains unclear.

Appendix E.4 reduces the labeled data used to combine votes, while keeping the selected programs fixed. It does not show whether the full method works well with few labels. The paper also does not report the total setup cost of tuning thresholds, selecting programs, and fitting the aggregator.

The paper does not report accuracy on the original test sets after editing the programs. Meanwhile, the rich-content bias win rate increases from 48.92% to 53.28% (Table 2), so the edits do not consistently reduce bias.

Questions for authors
  1. How does PAJAMA compare with a simple classifier using similar text features and the same labeled data, in terms of accuracy and speed?
  2. Can the programs detect factual or reasoning errors when answers have similar length and formatting?
  3. After editing the programs, how do accuracy and coverage on the original test sets change? Why does the rich-content bias win rate increase?
7 comments
AuthorsOpenReview ↗

Reviewer 3dPa (1/2)

Thank you for your thoughtful feedback! We are glad you found the paper clear and easy to follow, as the other reviewers also noted! We also appreciate you highlighting PAJAMA's competitive performance: it matches OLMo-2-13B-Inst. while running 47× faster. When combined with LLMs, the hybrid evaluation improves accuracy at higher throughput. Finally, reward models trained on PAJAMA labels outperform those trained on proprietary-model labels, at 45–50× lower API cost.

  • On Model-based Evaluator.
    • We thank the reviewer for suggesting this comparison. We further fine-tune ModernBERT as a pairwise evaluator using exactly the same supervision PAJAMA receives: each dataset's validation split. We train for 5 epochs over 3 seeds and report test accuracy. We also train one evaluator on all five validation splits pooled (a much stronger evaluator due to more supervision). Results are shown in the table below.
    • With the same labels, PAJAMA outperforms the fine-tuned evaluator on every dataset, by 14.9 points on average over the stronger pooled variant, while also being 1.7× faster without a GPU. Generated programs not only achieve higher throughput and performance competitive with LLM judges, but also outperform specialized evaluators that require potentially much more supervision. Moreover, programmatic judges are cheap to obtain and offer transparent decision logic to audit.
DatasetModernBERT (individual val)ModernBERT (pooled val)PAJAMA
JudgeLM62.5 ± 3.361.3 ± 6.181.1
PandaLM47.9 ± 0.455.2 ± 4.270.4
MultiPref50.1 ± 0.158.6 ± 5.872.2
Prometheus69.7 ± 9.373.4 ± 12.688.8
Preference 700k55.0 ± 1.557.8 ± 4.567.8
Avg. Acc. (%)57.061.276.1
Throughput (samples/sec)262262439
  • On Judging Correctness.

    • First, our fallback mechanism handles cases that programs cannot capture: samples that are uncertain or require deeper verification are routed to an LLM judge. When programs disagree or are uncertain, as is typical for subtle factual or reasoning errors that leave no easy-to-check trace, the committee's confidence drops, and these samples are sent to the LLM judge. In our results, routing only the 42% least-confident samples outperforms OLMo-2-13B-Instruct by 1.32 points while delivering 2.1× higher throughput.
    • Second, PAJAMA can also be easily extended with external knowledge (e.g., textbooks or encyclopedias carrying domain knowledge). Since programs are ordinary Python functions, they can call external tools, such as search and retrieval over a knowledge base to check factual claims, or code and math executors to verify reasoning steps. Such knowledge-augmented programs would plug directly into our committee and aggregation framework. Moreover, we offer a concrete example showing that our programs capture substance beyond length and formatting, as shown in our response to Reviewer TPKF.
  • On Effect of Validation Set Size.

    • Thank you for the question! To evaluate the full method under limited supervision, we re-ran the entire pipeline using only a subsample of the validation labels. We vary the validation set from 100% down to 5% over 5 random seeds. PAJAMA remains robust with fewer labels. With 20% of the labels, average accuracy drops by only 0.9 points, and even with 5% the drop is only 2.3 points. Notably, PAJAMA with just 25 labels per dataset still outperforms the ModernBERT evaluator fine-tuned on all pooled validation labels by 13 points.
Val labels usedJudgeLMPandaLMMultiPrefPrometheusPreference 700kAvg. Acc.
100%81.0 ± 0.071.8 ± 0.772.3 ± 0.189.4 ± 0.167.9 ± 0.176.5
80%81.0 ± 0.270.1 ± 0.673.2 ± 0.889.1 ± 0.768.9 ± 0.976.4
60%80.8 ± 0.370.0 ± 1.072.7 ± 0.689.1 ± 0.468.5 ± 1.276.2
40%80.8 ± 0.470.2 ± 0.772.7 ± 1.388.9 ± 0.867.5 ± 0.576.0
20%80.7 ± 0.869.5 ± 1.071.8 ± 1.887.7 ± 1.168.2 ± 0.975.6
5%80.0 ± 1.769.4 ± 1.570.1 ± 1.585.7 ± 4.065.8 ± 3.874.2
AuthorsOpenReview ↗

Reviewer 3dPa (2/2)

  • On Total Setup Cost.
    • We thank the reviewer for raising this and we are happy to offer its cost. PAJAMA has two phases: (i) an offline phase covering program synthesis, threshold tuning, program selection, and aggregator fitting, and (ii) an online phase covering program execution and aggregator inference. Our paper already reports online efficiency as throughput. Below, we report the runtime of each offline step after program synthesis. The full setup finishes in at most 15 seconds per dataset on a single CPU machine, with no GPU required. The total setup cost is therefore very low.
Setup stepJudgeLMPandaLMMultiPrefPrometheusPreference 700k
Normalize scores0.080.080.020.080.07
Tune thresholds0.170.160.160.160.16
Program selection14.6614.0711.8113.5113.64
Fit aggregator0.120.110.110.100.11
Total (sec)15.0214.4212.1013.8513.99
  • On Accuracy after Program Editing.
    • We thank the reviewer for this important question. During the rebuttal, we first run two more rounds of program editing with Claude Opus 4.6. In each round, Opus 4.6 audits and edits the programs from the previous round using the same calibration prompt. Then we evaluate the edited programs on the original Prometheus test set. Calibration substantially reduces bias at a small cost in accuracy: after three rounds, the average flip rate drops by 73%, while test accuracy decreases by only 1.6 points. Program calibration therefore makes programmatic judges more robust to bias while preserving accurate verdicts. This also highlights another advantage of programs: their evaluation logic is explicit and auditable, so biased features can be found and fixed directly. In contrast, for any model-based evaluator, it is hard to inspect the weights, locate a biased feature, and correct it directly.
MethodRich Content FR / BWR ↓Reference FR / BWR ↓Gender FR / BWR ↓Verbosity FR / BWR ↓Average FR / BWR ↓Test Acc. ↑Coverage
Round 0 (uncalibrated, in paper)23.13 / 48.9220.30 / 47.107.69 / 46.279.34 / 16.8415.12 / 39.7888.895.9
Round 1 (in paper)13.74 / 53.2817.29 / 43.173.70 / 36.173.25 / 9.689.50 / 35.5887.595.8
Round 2 (new)10.37 / 47.792.96 / 44.932.99 / 37.962.23 / 8.204.64 / 34.7287.195.9
Round 3 (new)6.77 / 44.032.96 / 44.204.48 / 39.422.23 / 8.204.11 / 33.9687.295.7

Thank you for the additional experiments. The ModernBERT comparison leaves open how much program synthesis adds over a lightweight classifier trained on the same labels and comparable text features.

RewardBench evaluates the downstream reward model trained on PAJAMA labels. When answers are matched for length and formatting, I’d still like to know how often the program committee itself confidently favors an answer with a factual or reasoning error.

Since both questions remain open, I’m keeping my original rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We are grateful that reviewers found our paper clearly written and easy to follow [3dPa, S8T2, TPKF], the problem interesting and our solution simple [S8T2], the key strength of O(n) v.s. O(1) [S8T2] and our framework a low-cost, high-throughput alternative to LLM judges, with routing that adds no inference cost [3dPa, S8T2, TPKF].

Below, we summarize our responses to the shared questions.

  • How does calibration affect bias and accuracy? We conducted two more program editing rounds, which reduce the average flip rate from 15.12 to 4.11, while maintain high test accuracy. We show that program logic is explicit and auditable, so biased features can be fixed directly.
  • How does PAJAMA compare with stronger baselines? With the same supervision, PAJAMA outperforms a fine-tuned ModernBERT evaluator by 14.9 points on average. Reward models trained on PAJAMA labels also outperform those trained on GPT-5.5 labels, at about 15× lower cost.
  • How robust is PAJAMA to fewer labels and new datasets? With 20% of validation labels, accuracy drops by only 0.9 points. Moreover, programs transfer across datasets with an average loss of 1.6 points, and offline setup takes at most 15 seconds per dataset on CPU.

Finally, we want to highlight the core idea that PAJAMA wants to bring to the field: it replaces repeated LLM inference calls with a set of programs that is synthesized once, runs locally, and can be reused in any future evaluation at no extra cost. Evaluation is not only about accuracy: latency, throughput, cost, and the ability to trace each judgment’s decision path also matter. As evaluation becomes more online and iterative, these limitations of LLM judges are hard to ignore. We studied PAJAMA from four angles: (1) programmatic judges on their own, (2) routing uncertain samples to LLM judges, (3) distilling reward models from programmatic judges, and (4) robustness to biased samples. Across all four, programmatic judges give fast, interpretable, and low-cost judgments with competitive accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.