- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PAJAMA, a method that generate simple Python scoring functions from evaluation rules and labeled examples. It combines their scores by a weak-supervision model and uses an LLM to judge difficult cases when the results are uncertain. The authors evaluated PAJAMA on five preference datasets, and compared it with different LLMs, and use its labels to train reward models. They also study how changes in the dataset affect bias. The standalone committee achieves 78.11% accuracy and processes 439.41 comparisons per second. This hybrid system offers a better balance between accuracy and speed. And the reward models that trained on labels from these Python programs also perform better on RewardBench than models trained on GPT-4 labels.
Critical strengths
1.It reframes repeated preference evaluation as an execution of reusable programs, it also addressed the cost of applying an LLM judge to each response pair. It combines those programs with the selective LLM fallback, making the efficiency gain useful even when the programs cannot handle every comparison reliably.
2.It makes its scoring rules as editable code, making the behavior of the evaluator easier to follow and revise. It also cost lesser when evaluating the reward-model training, showing that the generated signals are valuable beyond agreement on a preference benchmark.
Critical weaknesses
1.It hasn’t confirmed whether its program can get the substance right independently of surface cues related to preference labels. 2.Under similar supervision, PAJAMA's contribution to synthetic judgment rules still yet to be resolved compared to simpler heuristic evaluators.
Questions for authors
1.What proof is there that PAJAMA can tell if something’s actually correct without relying on how it looks or is formatted?
2.Why is program synthesis better than just using simple rule-based judges when you have the same amount of resources?
3.How well does the routing system catch program mistakes when you switch up the type of problems it’s evaluating?
4 comments
Reviewer TPKF (2/2)
- On Simpler Heuristics.
- In our preliminary work, we explored this idea extensively: we brainstormed many heuristic rules and evaluated them on the preference datasets. As a concrete example, we compare a length-based heuristic against the best single program synthesized by Claude Opus 4.6. The best program encodes far more complex decision logic (e.g., bigram/trigram redundancy, root type-token ratio, and sentiment markers) and handles edge cases that humans cannot easily write down. Writing such rules is cognitively demanding and time-consuming even for human experts, since it requires a deep understanding of each dataset. Most importantly, each condition must be tuned iteratively by hand to get it right. Even so, a single program cannot reliably cover all samples. This is why PAJAMA draws on collective expertise: it distills the LLM's judgment into a diverse committee of programs across evaluation rubrics and aggregates their verdicts into a final decision.
- We also measured the size of the selected programs. As shown in the table below, each program contains about 295 lines of code on average (up to 410), and each dataset's committee totals 2.4K–6.3K lines of evaluation logic. Writing and maintaining rules at this scale by hand for every new dataset would be impractical, whereas PAJAMA synthesizes them automatically.
Simpler heuristics vs. synthesized programs (test accuracy, %; coverage in parentheses).
| Dataset | "Longer response wins" | Best single program | PAJAMA |
|---|---|---|---|
| JudgeLM | 49.5 | 89.2 (50% cov.) | 81.1 (99% cov.) |
| PandaLM | 67.2 | 73.7 (65% cov.) | 70.4 (89% cov.) |
| MultiPref | 74.0 | 79.3 (50% cov.) | 72.2 (96% cov.) |
| Prometheus | 65.0 | 93.5 (55% cov.) | 88.8 (96% cov.) |
| Preference-700K | 66.7 | 73.3 (57% cov.) | 67.8 (96% cov.) |
| Avg. Perf. | 64.5 | 81.8 (55% cov.) | 76.1 (95% cov.) |
Synthesized programs are large: about 295 lines each on average.
| Dataset | Selected programs | Lines per program, mean (min–max) | Committee total lines |
|---|---|---|---|
| JudgeLM | 21 | 300 (233–373) | 6,297 |
| PandaLM | 14 | 265 (161–337) | 3,709 |
| MultiPref | 16 | 309 (242–410) | 4,939 |
| Prometheus | 8 | 305 (251–344) | 2,438 |
| Preference-700K | 15 | 294 (223–359) | 4,411 |
- On Routing Under a Change of Problem Type.
- We thank the reviewer for this question. We answer this through two experiments, using all 20 cross-dataset pairs (programs built for one dataset, applied to another).
- (i) Programs transfer. In Appendix E.5, we apply each dataset's programs to every other dataset without re-synthesis. Programs lose only 1.6 accuracy points on average; 15 of 20 pairs stay within 2 points of in-domain programs, and 4 pairs even match or exceed them. Synthesized programs capture largely reusable judging criteria: a program library built once can serve new datasets with only light recalibration.
- (ii) The router still catches program mistakes. In this new experiment, we send the 20% least-confident samples to the LLM judge and count how many of the committee's errors they contain. Random routing would catch 20%. We test two cases:
- Light refit (re-running only thresholds, selection, and aggregator on the new data; ~15 s): the router catches 32.8% of errors, close to the in-domain 35.7%.
- No refit (the whole pipeline applied unchanged): the router still catches 27.1%, still well above random.
- Overall, we show that our synthesized programs generalize well to new datasets and problem types, and with a quick recalibration on the new data, our router remains nearly as effective as in-domain.
- We thank the reviewer for this question. We answer this through two experiments, using all 20 cross-dataset pairs (programs built for one dataset, applied to another).
Synthesized programs transfer across datasets.
| Programs from ↓ / Evaluated on → | JudgeLM | PandaLM | MultiPref | Prometheus | Preference-700K | Avg. change |
|---|---|---|---|---|---|---|
| JudgeLM | 82.1 | 70.1 (−2.0) | 72.3 (−0.8) | 89.6 (−1.4) | 68.1 (−0.7) | −1.2 |
| PandaLM | 82.3 (+0.2) | 72.1 | 71.0 (−2.1) | 89.2 (−1.8) | 69.0 (+0.2) | −0.9 |
| MultiPref | 80.4 (−1.7) | 71.4 (−0.7) | 73.1 | 86.9 (−4.1) | 68.9 (+0.1) | −1.6 |
| Prometheus | 80.9 (−1.2) | 70.9 (−1.3) | 73.5 (+0.4) | 91.0 | 68.4 (−0.4) | −0.6 |
| Preference-700K | 75.2 (−6.9) | 71.0 (−1.1) | 71.3 (−1.8) | 85.8 (−5.2) | 68.8 | −3.8 |
The router still catches program mistakes on new datasets.
| Router | Data | % of committee errors caught ↑ | vs. random |
|---|---|---|---|
| Random | — | 20.0% | 1.00× |
| PAJAMA | Same dataset (in-domain) | 35.7% | 1.79× |
| PAJAMA | New dataset, light refit | 32.8% | 1.64× |
| PAJAMA | New dataset, no refit | 27.1% | 1.36× |
Dear Reviewer TPKF,
To further address your question about simpler heuristics, we ran a new experiment. We train logistic regression (LR) on the same validation split that PAJAMA uses, with two feature sets: 8 handcrafted text features and unigram TF-IDF features.
- Handcrafted text features. We use the 8 heuristic listed below. Each feature is computed for both responses, and the classifier receives the difference between the two scores, so it learns which response has more or less of each feature. This setup gives a direct comparison between simpler features and features distilled into programs.
- TF-IDF. We also use unigram TF-IDF features, trained the same way.
For each setting, we train LR over 5 trials and report the mean and standard deviation of test accuracy.
| # | Feature | What it measures | Rationale |
|---|---|---|---|
| 1 | chars | Number of characters | Length (longer answers are often preferred) |
| 2 | words | Number of words | Length; together with chars, also captures word length |
| 3 | sentences | Number of sentences | Amount of content |
| 4 | avg_sent_len | Words per sentence | Writing style (dense vs. terse) |
| 5 | bullets | Number of bullet lines (-, *, •) | Formatting |
| 6 | headers | Number of markdown headers (#) | Formatting |
| 7 | digits | Number of digit characters | Specificity (numbers, dates, steps) |
| 8 | urls_citations | Number of URLs and citation markers (e.g., [1], "et al.") | Apparent credibility |
New results (test accuracy, %)
| Dataset | Text Features + LR | TF-IDF (1-gram) + LR | PAJAMA |
|---|---|---|---|
| JudgeLM | 61.3 ± 0.5 | 78.8 ± 0.4 | 81.1 |
| PandaLM | 51.4 ± 7.8 | 56.1 ± 0.4 | 70.4 |
| MultiPref | 63.8 ± 1.0 | 65.7 ± 1.2 | 72.2 |
| Prometheus | 75.0 ± 0.7 | 72.2 ± 0.2 | 88.8 |
| Preference-700K | 66.4 ± 0.2 | 63.9 ± 0.6 | 67.8 |
| Average | 63.6 ± 1.6 | 67.3 ± 0.3 | 76.1 |
PAJAMA achieves the best accuracy on every dataset and the best average. On average, it outperforms TF-IDF + LR by 8.8 points and handcrafted features + LR by 12.5 points. These results show the value of program distillation: it draws on the reasoning and coding abilities of current LLMs to produce richer evaluation signals than either hand-designed heuristics or bag-of-words features. The heuristic features mainly capture surface-level properties such as length and formatting, and TF-IDF captures only lexical cues. In contrast, PAJAMA's programs encode the judging logic of LLM evaluators, which lets PAJAMA go beyond these surface signals and generalize better.
We also have a new experiment that show how programmatic judges apply on factuality datasets. We use the Factuality subset of RewardBench-2, where one response in each pair contains a factual error. We keep only pairs whose two responses are matched in length, so that length cannot serve as a shortcut. We measure how many pairs the program committee judges on its own, how accurate it is on those pairs, and how well the resulting hybrid performs. The program committee evaluates each pair and routes the pairs it is least confident about to an LLM judge. We pair it with three LLM judges: OLMo-2-13B-Instruct, Qwen2.5-14B-Instruct, and Gemma-3-12B-Instruct.
| Setting | Program correct (%) | Program wrong (%) | OLMo-2-13B (Acc. / Thr.) | Qwen2.5-14B (Acc. / Thr.) | Gemma-3-12B (Acc. / Thr.) |
|---|---|---|---|---|---|
| LLM only (route 100%) | — | — | 52.0 / 4.8 | 71.4 / 4.4 | 57.7 / 4.7 |
| Route 10% | 67.3 | 32.7 | 66.1 (+14.1) / 22.3 (4.6×) | 69.0 (−2.4) / 20.4 (4.7×) | 65.3 (+7.6) / 21.8 (4.6×) |
| Route 20% | 71.6 | 28.4 | 67.7 (+15.7) / 14.3 (3.0×) | 72.2 (+0.8) / 13.0 (3.0×) | 64.9 (+7.2) / 13.9 (3.0×) |
| Route 30% | 76.0 | 24.0 | 64.9 (+12.9) / 9.4 (1.9×) | 72.2 (+0.8) / 8.5 (1.9×) | 63.3 (+5.6) / 9.1 (1.9×) |
| Route 40% | 78.2 | 21.8 | 59.7 (+7.7) / 6.8 (1.4×) | 71.4 (0.0) / 6.2 (1.4×) | 58.9 (+1.2) / 6.7 (1.4×) |
Program correct / wrong: accuracy and error rate on the pairs the committee labels itself. Acc.: hybrid accuracy (%); Thr.: throughput (samples/sec).
On length-matched factuality pairs, the hybrid with 20% routing outperforms each of the three 12–14B LLM judges used alone, at about 3× their throughput. This shows that programs complement LLM judges, and that they assess samples by more than surface cues or formatting features. As more pairs are routed, the committee’s error rate on the pairs it keeps drops steadily, from 32.7% to 21.8%. This suggests that its confidence scores reliably identify the cases it cannot handle and hand them off to the LLM judge.
Dear Reviewer TPKF,
As the discussion period is drawing to a close, we wanted to follow up on your questions. In addition to our initial response, we have run two new experiments to address your concerns.
These results show: (i) how decision paths as a evidence can be presented to users; (ii) the direct benefit of program distillation over simpler heuristics; (iii) improved performance on factuality and reasoning datasets; and (iv) how the routing system generalizes to different types of questions. We would be glad to hear your thoughts and are happy to answer any further questions.
Thank you for your time and for your valuable feedback.
Reviewer TPKF (1/2)
Thank you for your positive review! We are glad you found the paper clear and easy to follow, as the other reviewers also noted. We appreciate your recognition of the key strengths of our work: low cost, high throughput evaluation. When combined with LLM judges, PAJAMA uses program outputs as routing signals to escalate uncertain samples, with no additional inference latency. Programs also make evaluation logic explicit and traceable, so current coding agents can calibrate them easily. Finally, reward models trained on program labels outperform those trained on proprietary-model labels, at 45–50× lower cost.
judge_41, 324 lines in total), which computes 12 weighted features targeting clarity and tone written by Opus 4.6.Given Sample
The program scores Response 1 at 3.36 and Response 2 at 8.11, matching the human preference. Notably, the preferred response is shorter, so a length heuristic would choose wrongly. This shows that synthesized programs capture judging logic beyond surface cues such as length. The decision is driven by interpretable signals:
Removing the flagged rubrics costs little accuracy (test accuracy, %).