acceptodds
Under review as a conference paper at ICLR 2027

Conv-to-Bench: Exploring Automated Benchmark Construction from Conversational Traces

Abstract

The continuous advancement of large language models requires robust, scalable benchmarking. Traditional benchmark creation relies on expensive expert curation, while prior automated methods focus primarily on single-turn interactions and rely heavily on proprietary models. We introduce Conv-to-Bench, a multi-stage framework that automatically transforms authentic human–LLM dialogues into self-contained instructions paired with fine-grained, verifiable requirement checklists using open-weight models for construction and judging. Moving beyond single-turn queries, our pipeline captures multi-turn user refinements to formulate realistic evaluation criteria. Across an aligned 19-model roster evaluated against Arena AI human preferences, Conv-to-Bench achieves comparable rank correlation with human judgments and Brier scores relative to Arena-Hard-Auto, a widely used automated benchmark (– vs.; Brier – vs.), without proprietary API reliance. Ablation experiments demonstrate that atomic checklist verification significantly outperforms pairwise comparisons across Spearman correlation (), separability ( pp), agreement ( pp), and Brier score (). Incorporating multi-turn refinements yields mixed results: Spearman correlation point estimates improve across all LLM judges, but significant gains in correlation and Brier score occur for only one individual judge. Conv-to-Bench builds a 500-instruction benchmark for USD 19.05 and evaluates candidate models for USD 1.62 each on hosted infrastructure. Together, these results show that open-weight models can build benchmarks that recover human preference rankings from conversational data, enabling their application in lower-budget settings.

Primary area:
datasets and benchmarks
Keywords:
LLM benchmarking, LLM evaluation, LLM-as-a-judge, checklist-based evaluation, large language models, multi-turn conversations, open-weight models, automated benchmark construction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TB3fOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Conv-to-Bench, a framework that builds benchmarks from real human-LLM conversations using only open-weight models. A bandit-based sampler and a multi-stage filter select useful dialogues. The pipeline then turns them into self-contained instructions, either from the first turn alone or merged with later user refinements, and pairs each with a checklist of atomic yes/no requirements. Models are scored by LLM judges against these checklists. The resulting 500-instruction benchmark ranks 19 models in close agreement with Arena AI human preferences, comparable to Arena-Hard-Auto, at a much lower cost. Ablations show that checklist verification clearly beats pairwise judging. Adding multi-turn refinements gives mostly non-significant gains.

Critical strengths

[1] The paper addresses a practical need. It shows that a benchmark can be built from existing conversation logs at low cost with open-weight models, without proprietary APIs. This makes the approach usable for organisations with smaller budgets, which is the main reason the work matters.

[2] The design choices are well motivated by the ablations. Checklist versus pairwise judging, and base versus refined instructions, show why each component is there. The bandit sampler and multi-stage filter are also important for curating useful conversations out of noisy logs.

Critical weaknesses

[1] The pipeline’s robustness to the underlying model is not shown. The prompts for instruction and checklist synthesis, and the filtering, depend on the specific open-weight models used (the gpt-oss family). The quality of the resulting benchmark could change considerably with other, cheaper models. The paper should include an experiment or discussion on whether the creation process holds up when the core model is swapped.

[2] The validation against human preferences is limited to one setting. The alignment in Section 4.3 uses a 19-model roster frozen at mid-2024, which excludes recent models. It should be repeated with a more recent checkpoint to see whether current proprietary models give similar results. The comparison with Arena-Hard-Auto also shows no statistically significant differences, so the claim is “comparable”, not better.

Nit: The labels in Figure 2 are hard to read.

Questions for authors

Is there a cost performance tradeoff? As in with more cheaper open-weight models, does the comparison with Arena Hard hold or does it degrade?

5 comments

We thank the reviewer for the constructive feedback and insightful comments on our work. Below, we address the specific points raised.

Weakness 2 - About the recency of the models

We evaluated 6 recent models (released in 2025–2026, 5 of them reasoning models): the benchmark keeps a high rank correlation with Arena AI and orders them close to the human-preference ranking (results below).

The selected model roster is relatively old because of the baseline benchmark generation method we compare our work to, which is Arena-Hard-Auto. Our baseline data relies on the results of models released by mid-2024 (Section 4.3; both references are frozen at 2024-07-31). To ensure a valid comparison, we pair evaluated models between the two methods. However, to test whether the generated benchmark also holds up when evaluating newer models, we chose 6 models released between mid-2025 and 2026, including 5 reasoning models, and evaluated them. Results are below.

ModelReasoningScore (95% CI)Arena AIRank in Arena AI
DeepSeek-V4.1-FlashYes85.9% [84.2; 87.5]1473.31
Muse Glimmer 30BYes84.6% [82.8; 86.3]1386.44
MiMo-V2.6-FlashYes82.2% [80.2; 84.0]1473.32
Step-3.5-FlashYes81.9% [80.1; 83.6]1416.93
Nemotron-3-Nano 30BYes79.8% [78.1; 81.5]1377.15
Mistral Small 3.2 24BNo74.3% [72.3; 76.2]1357.76

Our benchmark preserves the Arena AI order of these models except for one (Muse Glimmer 30B, ranked 4th by Arena AI and 2nd by our benchmark), and when combining these 6 with the older 19 models, the ranking (gemma4-31b as judge) shows high correlation (Spearman 0.970 [0.874; 0.996]), agreement (86.8%) and separability (86.0%).

The results indicate that the newer models are slightly saturated on our benchmark, but this can be explained by the date of the conversations used to build it, which reflect older use cases. Despite this minor ceiling effect, the benchmark distinguishes these recent models from the 2024 roster and maintains high correlation in ranking. We will add these results to the revised manuscript. We agree with the reviewer that the comparison with Arena-Hard-Auto supports a “comparable” rather than a “better” claim, which is how we state it in Section 5.1 and Section 6. We also note that our primary contribution is the method of generating benchmarks and not the generated benchmark itself.

Question 1 - About cost-performance tradeoff

We believe that is the case, superior open-weight models are likely to be better at understanding user requirements and formulating instructions and checklists, so cheaper models could degrade the comparison with Arena-Hard-Auto. Our pipeline already uses the smaller gpt-oss-20b for the filtering stage (Section 4.1). However, the use of open-weight models itself for this task is already a novel contribution, and we leave a deeper exploration of this tradeoff for future work.

Weakness 1 - About the pipeline's robustness to the underlying model

We are running an experiment to evaluate this robustness. As soon as the results are available, we will post them on this thread.

AuthorsOpenReview ↗

Robustness of the pipeline to the underlying model

Swapping the Stage 2 synthesis model to gemma-4-31B-it, a ~4× smaller model from a different family, without prompt tuning preserves and even improves human-rated quality (+0.28 instruction mean and -6.0 pp checklist defects), confirming that our pipeline is robust and not tied to gpt-oss-120b.

Following up on our previous response, below are the full results of swapping the Stage 2 synthesis model.

Sample & Protocol. Using a stratified random sample of 50 dialogues from the 500-task benchmark (seed 42; preserving the multi-turn share), we re-ran the full instruction and checklist synthesis stage (Stage 2) replacing gpt-oss-120b with gemma-4-31B-it, a model from a different family with roughly 4× fewer parameters, while keeping all prompts unchanged. Each model generated one instruction–checklist pair per dialogue under the benchmark rule (refined if valid, base otherwise). Two annotators blinded to the generator evaluated all 50 pairs for both models after reading the full source dialogue. The same annotation also serves as the human validation of the benchmark's instructions and checklists (see our response to Reviewer Gymm). We used a crossed design where each annotator rated 25 pairs per model and saw each dialogue once, preventing annotator effects from biasing the comparison.

Rubric. Derived directly from the Stage 2 prompts:

  • Instruction (1–5 scale per criterion):
    • I1 Clarity: Unambiguous, admitting a single interpretation.
    • I2 Self-containment: Solvable from the instruction alone.
    • I3 Feasibility: Fulfillable purely via text generation.
    • I4 Fidelity: Accurately reflects user intent (including later refinements for refined instructions) without omitting or fabricating requirements.
  • Checklist (3 items sampled uniformly per checklist to avoid revealing model identity via length; binary defect flags):
    • D1 Not entailed: Requires something neither stated nor implied by the instruction.
    • D2 Not verifiable: Cannot be judged objectively yes/no.
    • D4 Incorrect: Factually wrong.

Results. (Δ = gemma-4-31B − gpt-oss-120b; 95% CIs via conversation-level bootstrap adjusted for annotator).

Criteriongpt-oss-120bgemma-4-31BΔ (95% CI)
I1 Clarity4.624.80+0.18 [-0.05; +0.43]
I2 Self-containment4.704.90+0.20 [-0.05; +0.50]
I3 Feasibility4.604.92+0.32 [+0.08; +0.60]
I4 Fidelity to the user4.344.74+0.40 [+0.09; +0.71]
Instruction mean4.574.84+0.28 [+0.06; +0.51]
D1 Not entailed13.3%5.3%-8.0 pp [-15.0; -1.4]
D2 Not verifiable1.3%2.0%+0.7 pp [-2.9; +3.8]
D4 Incorrect0.0%1.3%+1.3 pp [+0.0; +4.4]
Any checklist defect14.7%8.7%-6.0 pp [-14.5; +2.0]

Takeaways.

  • Cross-family prompt transferability (Weakness 1): Swapping to a different model family without prompt tuning preserves (and slightly improves) human-rated quality across criteria (+0.28 instruction mean; -8.0 pp non-entailed checklist items).
  • Model-agnostic quality bar: Under both generators, incorrect and unverifiable checklist items remain ≤ 2% and instruction fidelity stays above 4.3/5.

Note that this experiment evaluates artifact synthesis quality rather than downstream Arena AI ranking correlation (which would require regenerating all 500 tasks and re-evaluating all 19 models, left for future work).

Dear Reviewer TB3f,

We would like to thank you again for the valuable feedback. The submission has been updated to address the points raised, including the new experiments. Please let us know if you have any remaining questions.

Best regards, The Authors.

Reviewer_VH2XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce Conv-to-Bench, an approach for converting real-world LLM interactions into a benchmark with test cases based on the original interactions along with judgement criteria. The approach consists of a (1) selection phase with topic clustering, bandit-based sampling and content filters (language, policy violations, etc.), (2) a checklist creation phase (which includes compressing multi-turn prompts in to a single one) and (3) a scoring phase where models are tested based on the tasks created in the previous two phases. The basic idea is to use single-model rather than the more typical pairwise assessments, as often done by LLM-as-a-Judge.

Contribution. The authors highlight the multi-turn user refinements, and atomic checklist verification, and reliance on open-weight models as their unique contribution. With Arena-Hard-Auto as a baseline.

Validation. The authors compare the rankings generated by their method based on WildChat/LMSys-Chat-1M to Chatbot Arena (Arena AI) rankings and Arena-Hard-Auto rankings. They also do some ablation experiments within their method, regarding use of checklist vs pairwise judgement and base vs refined prompts.

Critical strengths
  1. Important problem. Coming up with good benchmarks that correlate with real-world user utility is an important challenge of benchmarking, and remains an ongoing challenge as model capabilities and use-cases evolve.
  2. Reproducible pipeline. The authors use open-source models and share their code as supplementary material. Compared to work that only relies on API-accessible models, this helps make the work reproducible in the long-run. Always good to see.
Critical weaknesses

By far the biggest weakness is the validation in two ways

  1. Outdated evaluation problem: The authors only test their framework's ability evaluating models that are over two years old (older than 2024-07-31). Since then model capabilities, and the aspects that make one model create more user utility than another have drastically changed (back then basic coding and math was far less solved). For example, no reasoning models are assessed, yet now every model is a reasoning model. The way I use LLMs today is very different to how I used them in 2024. Benchmarks are inherently very time-sensitive, their goal should be to evaluate current and future AI models, the paper is not demonstrating that ability based on their experiments.
  2. Too many factors varied: The authors' contribution is a pipeline, but in their comparisons against baselines (human preferences, Arena-Hard-Auto) they vary the pipeline as well as the source data and the judge model (relative to Arena-Hard-Auto). This setup does not allow the reader to isolate the effect of their contribution, the pipeline!

Unfortunately the weaknesses clearly outweigh the strengths for me

Questions for authors
  1. Have you tested random sampling vs the bandit-based sampling? It seems a bit arbitrary and complicated as part of the pipeline, it's not clear the bandit-based sampling is necessary or useful.

Further suggestions (no need to reply, do not affect score)

  1. Fig 2 is way to small, can't read
7 comments

We thank the reviewer for their thoughtful feedback and the time dedicated to evaluating our submission. Below, we address the specific points raised.

Weakness 1 - About the recency of the models

We evaluated 6 recent models (released in 2025–2026, 5 of them reasoning models): the benchmark keeps a high rank correlation with Arena AI and orders them close to the human-preference ranking (results below).

The selected model roster is relatively old because of the baseline benchmark generation method we compare our work to, which is Arena-Hard-Auto. Our baseline data relies on the results of models released by mid-2024 (Section 4.3; both references are frozen at 2024-07-31). To ensure a valid comparison, we pair evaluated models between the two methods. However, to test whether the generated benchmark also holds up when evaluating newer models, we chose 6 models released between mid-2025 and 2026, including 5 reasoning models, and evaluated them. Results are below.

ModelReasoningScore (95% CI)Arena AIRank in Arena AI
DeepSeek-V4.1-FlashYes85.9% [84.2; 87.5]1473.31
Muse Glimmer 30BYes84.6% [82.8; 86.3]1386.44
MiMo-V2.6-FlashYes82.2% [80.2; 84.0]1473.32
Step-3.5-FlashYes81.9% [80.1; 83.6]1416.93
Nemotron-3-Nano 30BYes79.8% [78.1; 81.5]1377.15
Mistral Small 3.2 24BNo74.3% [72.3; 76.2]1357.76

Our benchmark preserves the Arena AI order of these models except for one (Muse Glimmer 30B, ranked 4th by Arena AI and 2nd by our benchmark), and when combining these 6 with the older 19 models, the ranking (gemma4-31b as judge) shows high correlation (Spearman 0.970 [0.874; 0.996]), agreement (86.8%) and separability (86.0%).

The results indicate that the newer models are slightly saturated on our benchmark, but this can be explained by the date of the conversations used to build it, which reflect older use cases. Despite this minor ceiling effect, the benchmark distinguishes these recent models from the 2024 roster and maintains high correlation in ranking. We will add these results to the revised manuscript. We also note that our primary contribution is the method of generating benchmarks and not the generated benchmark itself.

Weakness 2 - About the isolation of factors in the comparisons

We agree with the reviewer’s observation that, as presented, our comparison against Arena-Hard-Auto varies the source data and judge model, not isolating the pipeline itself. This comparison was intended to evaluate our end-to-end framework as a complete method against the baseline as a whole, capturing the combined effect of data sampling, instruction synthesis, and the evaluation protocol. The individual factors are examined separately, as described below.

First, we ablated two key factors within our own pipeline, holding all other conditions constant. We isolated (i) the effect of multi-turn interactions (base vs. refined; Section 5.3, Table 3) and (ii) the evaluation mechanism itself, through an experiment that holds the source data, judge model (GLM-4.7-Flash), instructions and responses constant, varying only the protocol: checklist verification vs. pairwise comparison (Section 5.2). This experiment was updated with a better-suited pairwise baseline following another reviewer’s comment (see our response to Reviewer Gymm). The two protocols reach statistically indistinguishable agreement with Arena AI, while checklist verification retains its structural advantages, such as item-level explainability and independence from a baseline model. The revised manuscript will report this updated version.

Regarding the other two factors, the divergence in source data is intentional, because one of the components of our framework is the bandit-based data selection mechanism, used to reduce the cost of navigating a broad corpus of conversations (see our answer to Question 1). As for the judge, we evaluated three open-weight checklist judges and their ensemble, which reach similar ranking correlations with Arena AI (Spearman 0.926–0.946; Table 1) and strong item-level consistency (Krippendorff’s α = 0.764; Section 5.4), indicating that the choice of judge has little effect on the final rankings.

In short, the two sets of experiments are complementary. Comparing against Arena-Hard-Auto shows how our framework performs as a complete method, and the ablations help to understand how each component affects the final metrics.

Question 1 - About random vs bandit-based sampling

Yes, with the same number of drawn conversations, the bandit-based sampler yields 1.41x more approved conversations than random sampling, at a lower token cost per approval. We had observed this trend in preliminary tests during development, but those runs were not at the scale needed for inclusion in the paper, so we ran the full-scale comparison in response to your question.

In this comparison, random sampling was run until it reached the same number of conversations processed by the bandit-based version (16,731 drawn conversations, Section 4.1). We compare the number of approved samples and cost per approval, since the main motivation for this mechanism is to reduce costs in this initial task of navigating a broad corpus of conversations.

The bandit-based sampler approved 539 conversations compared to 383 from the uniform random baseline. Consequently, the bandit mechanism's token cost per approved conversation was 1.34x lower (118.8 thousand tokens versus 159.5 thousand tokens).

Regarding topical diversity, with the same number of drawn conversations, the conversations approved by the bandit cover more topic clusters than those from random sampling (147 vs. 119), since more conversations are approved. Normalized by the number of approvals, diversity is slightly lower for the bandit (27.3 vs. 31.1 clusters per 100 approved conversations), so the higher yield comes with only a small reduction in diversity per approval.

The bandit-based approach therefore yields more approved conversations, covering more topics, at a lower cost per approval, which supports its use in the pipeline.

We will also enlarge Fig. 2 to improve readability.

Thank you for getting back to me with your response!

high correlation (Spearman 0.970 [0.874; 0.996]), agreement (86.8%) and separability (86.0%).

please also report these metrics for only the 6 new(er) models, and also do the deepseek and mimo models really have exactly the same arena score or is this an error?

Weakness 2 - About the isolation of factors in the comparisons

Without a direct comparison of pipelines with fixed corpus and judge, there is not sufficient evidence to support the paper's main claim that this pipeline adds value.

Reviewer_GymmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Conv-to-Bench, an automated framework that converts real human–LLM conversations into self-contained evaluation tasks and fine-grained requirement checklists using open-weight models. The framework incorporates user refinements from later dialogue turns and uses LLM judges to score model responses against the generated checklists. The authors construct 500 English tasks from WildChat and LMSYS-Chat-1M and evaluate 19 models using three open-weight judges. The resulting rankings show high correlations with Arena AI human-preference rankings, although the differences from Arena-Hard-Auto are not statistically significant. Ablation experiments show that checklist-based scoring outperforms the tested pairwise evaluation protocol, while the benefits of incorporating multi-turn refinements are less consistent. The authors also report low benchmark-construction and evaluation costs.

Critical strengths
  1. The framework provides a low-cost approach to automatically constructing evaluation benchmarks from real user conversations using open-weight models. The reported construction and evaluation costs suggest practical advantages over existing approaches.
  2. The checklist-based scoring method achieves significantly better performance than the tested pairwise baseline on several evaluation metrics. The experiments also include multiple LLM judges and ablation analyses, providing useful evidence on the framework's effectiveness.
Critical weaknesses
  1. The evaluation is limited to 19 models released by mid-2024. In Table 2, the pairwise baseline also uses a single, much newer model (Qwen3.6-27B) as the fixed reference, which may make it difficult to distinguish weaker candidate models. It remains unclear whether checklist scoring would still outperform pairwise evaluation with different reference models or when evaluating more recent models with similar capabilities. This limits the evidence supporting one of the paper's main findings.
  2. The paper mainly validates the generated benchmark through its correlation with Arena AI rankings. However, this does not directly establish whether the generated instructions faithfully preserve user requirements or whether the checklists provide accurate evaluation criteria. The lack of independent human validation raises concerns about the reliability of the constructed benchmark, which is central to the paper's contribution.
Questions for authors
  1. Could the authors evaluate checklist-based scoring against pairwise evaluation using reference models with different capability levels? Reporting the win/loss/tie distributions against Qwen3.6-27B would also help determine whether the current pairwise baseline is affected by one-sided comparisons.
  2. Could the authors extend the evaluation to more recent models, particularly those with similar capabilities? This would help assess whether the proposed benchmark can reliably distinguish stronger models beyond the 19 models used in the current experiments.
  3. Could the authors provide human validation of a representative subset of generated instructions, checklists, and grading results? In particular, it would be useful to examine whether the extracted instructions preserve the original user requirements and whether the generated checklists provide appropriate evaluation cri
5 comments

We sincerely thank you for your time, constructive feedback, and valuable insights. Your comments have significantly helped us improve the quality and clarity of our work. Below, we address the specific points raised.

Weakness 1 / Question 2 - About the recency of the evaluated models

We evaluated 6 recent models (released in 2025–2026, 5 of them reasoning models): the benchmark keeps a high rank correlation with Arena AI and orders them close to the human-preference ranking (results below).

The selected model roster is relatively old because of the baseline benchmark generation method we compare our work to, which is Arena-Hard-Auto. Our baseline data relies on the results of models released by mid-2024 (Section 4.3; both references are frozen at 2024-07-31). To ensure a valid comparison, we pair evaluated models between the two methods. However, to test whether the generated benchmark also holds up when evaluating newer models, we chose 6 models released between mid-2025 and 2026, including 5 reasoning models, and evaluated them. Results are below.

ModelReasoningScore (95% CI)Arena AIRank in Arena AI
DeepSeek-V4.1-FlashYes85.9% [84.2; 87.5]1473.31
Muse Glimmer 30BYes84.6% [82.8; 86.3]1386.44
MiMo-V2.6-FlashYes82.2% [80.2; 84.0]1473.32
Step-3.5-FlashYes81.9% [80.1; 83.6]1416.93
Nemotron-3-Nano 30BYes79.8% [78.1; 81.5]1377.15
Mistral Small 3.2 24BNo74.3% [72.3; 76.2]1357.76

Our benchmark preserves the Arena AI order of these models except for one (Muse Glimmer 30B, ranked 4th by Arena AI and 2nd by our benchmark), and when combining these 6 with the older 19 models, the ranking (gemma4-31b as judge) shows high correlation (Spearman 0.970 [0.874; 0.996]), agreement (86.8%) and separability (86.0%).

The results indicate that the newer models are slightly saturated on our benchmark, but this can be explained by the date of the conversations used to build it, which reflect older use cases. Despite this minor ceiling effect, the benchmark distinguishes these recent models from the 2024 roster and maintains high correlation in ranking. We will add these results to the revised manuscript. We also note that our primary contribution is the method of generating benchmarks and not the generated benchmark itself.

Weakness 2 / Question 1 - About the baseline choice on pairwise evaluation

With a mid-capability baseline, there is no significant difference on any metric comparing pairwise and checklist evaluation, while checklist keeps the advantages listed at the end of this section. Against the original Qwen3.6-27B baseline, the evaluated models lost 96.1% of the matches, securing only 2.2% wins and 1.7% ties, which compressed the pairwise scores at the floor. To test this, we re-ran the pairwise evaluation with a different baseline model: Llama-3-8B-Instruct, which is much closer to the middle of the evaluated models in terms of performance, as shown by the Arena AI reference ranking (position 13 out of 19). We evaluated the remaining 18 models together with the 6 newer models from our answer to Weakness 1 (24 models), using the same GLM-4.7-Flash judge (comparing Checklist against Pairwise BT with style control):

MetricChecklistPairwise BTDifference (95% CI)
Spearman0.9550.956-0.001 [-0.062; +0.081]
Separability85.1%89.1%-4.0 pp [-12.0; +1.4]
Brier ↓0.0430.058-0.015 [-0.057; +0.023]

Note that the pairwise evaluation now holds its own against checklist: the two methods are statistically tied on all metrics (every paired bootstrap 95% CI includes zero). However, because the newer models are stronger, the Llama-3-8B baseline becomes too weak for them, saturating the pairwise win rates in the opposite direction (the new models win 94.5% of their matches).

Checklist evaluation still offers several advantages over pairwise evaluation, including robustness to well-documented biases of LLM judges such as position and verbosity bias (Zheng et al., 2023):

  • Explainability: unlike pairwise judgments, which produce a scalar preference (A vs. B) with no guarantee of consistent criteria across comparisons, checklist evaluation provides explicit, inspectable criteria. This makes it possible to audit the exact requirements and verify how individual responses satisfied them, making failure modes transparent.
  • Baseline independence and longevity: as the results above show, the pairwise method's performance varies considerably depending on the choice of baseline model. As newer models are evaluated, the baseline must be changed to keep the rankings valid, which requires re-evaluating all models. Checklist evaluation therefore has greater longevity, as it does not depend on a baseline model.
  • No position bias: in the pairwise evaluation above (Llama-3-8B baseline, 24 models), swapping the order of the two responses changed the winner in 16.6% of the pairs, and models won 11.5 pp more often when shown first. The bias is strongest for models close to the baseline, which are precisely the comparisons the ranking needs to resolve. Checklist evaluates a single response at a time, avoiding this bias by construction.
  • No detectable self-preference bias: LLM judges are known to favor their own generations, in both pairwise and single-response judgments (Panickssery et al., 2024). Among our checklist judges, gemma-4-31B-it evaluates five models of its own family (Gemma 1, 1.1 and 2). A difference-in-differences analysis against the other two judges showed a -0.5 pp difference for these models, meaning there is no detectable preference for its own family. Since these are earlier models of the family rather than the judge itself, this is a family-level check.
  • No verbosity bias: on the pairwise evaluation above (Llama-3-8B baseline, 24 models, same GLM-4.7-Flash judge and same responses), the model-centered correlation between response length and score is slightly negative for checklist (Spearman -0.033) but positive for pairwise (+0.074). The style-controlled Bradley-Terry regression confirms this: its length coefficient is positive (0.18, 95% CI [0.14; 0.21]), meaning the pairwise judge favors the response that is longer than the baseline's, which is why pairwise requires style control. Checklist does not reward models for simply writing more.

The revised manuscript will replace the pairwise ablation with this updated version and include the position, self-preference and verbosity bias analyses.

Question 3 - About human validation

We are running a human evaluation of the generated instructions and checklists to ensure their quality. As soon as the results are in, we will post them on this thread.

References

Zheng, L., Chiang, W.-L., Sheng, Y., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.

Panickssery, A., Bowman, S. R., and Feng, S. LLM Evaluators Recognize and Favor Their Own Generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024.

AuthorsOpenReview ↗

Human evaluation of instructions and checklists

Our human validation study on 50 sampled tasks confirms that generated instructions achieve a 4.57/5 mean quality score with high intent fidelity (4.34/5), while 85.3% of checklist items are entirely defect-free (0% factually incorrect).

Below are the full results of our human validation study on the generated instructions and checklists.

Sample & Protocol. We drew a stratified random sample of 50 dialogues from the 500-task benchmark (seed 42; 29 refined, 21 base instructions). Two annotators, each rating 25 of the pairs, evaluated the instruction–checklist pairs after reading the full source dialogue. Annotators were blinded to the generator model (the study also evaluated a second generator; see our response to Reviewer TB3f) and could skip uncertain pairs. All 50 received complete ratings.

Rubric. Criteria match the exact Stage 2 generation prompts:

  • Instruction (1–5 scale per criterion):
    • I1 Clarity: Unambiguous, admitting a single interpretation.
    • I2 Self-containment: Solvable from the instruction alone, with all required context/artifacts included.
    • I3 Feasibility: Fulfillable purely via text generation.
    • I4 Fidelity: Accurately reflects user intent without omitting or fabricating requirements (judged against the first-turn request for base instructions, and against the user's final intent, including later refinements, for refined instructions, without leaking the solution).
  • Checklist (3 items sampled uniformly per checklist; binary defect flags):
    • D1 Not entailed: Requires something neither stated nor implied by the instruction.
    • D2 Not verifiable: Cannot be judged objectively yes/no (subjective or compound).
    • D4 Incorrect: Factually wrong or degrades response quality.

Results.

Instruction criterionMean (1–5)Score ≥ 4Score ≤ 2
I1 Clarity4.6290%4%
I2 Self-containment4.7090%8%
I3 Feasibility4.6086%4%
I4 Fidelity to the user4.3480%12%
Mean4.57——

Overall, 7 of 50 instructions (14%) scored ≤ 2 on at least one criterion.

Checklist defectShare of evaluated items
D1 Not entailed13.3%
D2 Not verifiable1.3%
D4 Incorrect0.0%
Any defect14.7% (22 / 150 items)

Defects occur mostly in base instructions (20.6% of items vs. 10.3% in refined instructions).

Takeaways:

  • Intent preservation: Fidelity averages 4.34/5 (80% scoring ≥ 4), directly confirming low rates of omitted/invented constraints or missed multi-turn refinements.
  • Checklist validity: 0% of sampled items were incorrect (D4), 1.3% were unverifiable (D2), and 85.3% were defect-free; remaining issues are minor over-specifications (D1), mostly on base instructions.
  • Self-containment: A 4.70 mean confirms tasks are solvable without the source conversation.

The human evaluation also helped us identify where the pipeline struggles. The annotators found more problems in longer, multi-turn conversations than in short ones, and in anomalous conversations, for example when the user pastes a log of an earlier conversation or a whole web page into the first turn. We will discuss these cases as limitations in the revised manuscript, along with possible mitigations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.