- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Conv-to-Bench, a framework that builds benchmarks from real human-LLM conversations using only open-weight models. A bandit-based sampler and a multi-stage filter select useful dialogues. The pipeline then turns them into self-contained instructions, either from the first turn alone or merged with later user refinements, and pairs each with a checklist of atomic yes/no requirements. Models are scored by LLM judges against these checklists. The resulting 500-instruction benchmark ranks 19 models in close agreement with Arena AI human preferences, comparable to Arena-Hard-Auto, at a much lower cost. Ablations show that checklist verification clearly beats pairwise judging. Adding multi-turn refinements gives mostly non-significant gains.
Critical strengths
[1] The paper addresses a practical need. It shows that a benchmark can be built from existing conversation logs at low cost with open-weight models, without proprietary APIs. This makes the approach usable for organisations with smaller budgets, which is the main reason the work matters.
[2] The design choices are well motivated by the ablations. Checklist versus pairwise judging, and base versus refined instructions, show why each component is there. The bandit sampler and multi-stage filter are also important for curating useful conversations out of noisy logs.
Critical weaknesses
[1] The pipeline’s robustness to the underlying model is not shown. The prompts for instruction and checklist synthesis, and the filtering, depend on the specific open-weight models used (the gpt-oss family). The quality of the resulting benchmark could change considerably with other, cheaper models. The paper should include an experiment or discussion on whether the creation process holds up when the core model is swapped.
[2] The validation against human preferences is limited to one setting. The alignment in Section 4.3 uses a 19-model roster frozen at mid-2024, which excludes recent models. It should be repeated with a more recent checkpoint to see whether current proprietary models give similar results. The comparison with Arena-Hard-Auto also shows no statistically significant differences, so the claim is “comparable”, not better.
Nit: The labels in Figure 2 are hard to read.
Questions for authors
Is there a cost performance tradeoff? As in with more cheaper open-weight models, does the comparison with Arena Hard hold or does it degrade?
5 comments
Robustness of the pipeline to the underlying model
Swapping the Stage 2 synthesis model to gemma-4-31B-it, a ~4× smaller model from a different family, without prompt tuning preserves and even improves human-rated quality (+0.28 instruction mean and -6.0 pp checklist defects), confirming that our pipeline is robust and not tied to gpt-oss-120b.
Following up on our previous response, below are the full results of swapping the Stage 2 synthesis model.
Sample & Protocol. Using a stratified random sample of 50 dialogues from the 500-task benchmark (seed 42; preserving the multi-turn share), we re-ran the full instruction and checklist synthesis stage (Stage 2) replacing gpt-oss-120b with gemma-4-31B-it, a model from a different family with roughly 4× fewer parameters, while keeping all prompts unchanged. Each model generated one instruction–checklist pair per dialogue under the benchmark rule (refined if valid, base otherwise). Two annotators blinded to the generator evaluated all 50 pairs for both models after reading the full source dialogue. The same annotation also serves as the human validation of the benchmark's instructions and checklists (see our response to Reviewer Gymm). We used a crossed design where each annotator rated 25 pairs per model and saw each dialogue once, preventing annotator effects from biasing the comparison.
Rubric. Derived directly from the Stage 2 prompts:
- Instruction (1–5 scale per criterion):
- I1 Clarity: Unambiguous, admitting a single interpretation.
- I2 Self-containment: Solvable from the instruction alone.
- I3 Feasibility: Fulfillable purely via text generation.
- I4 Fidelity: Accurately reflects user intent (including later refinements for refined instructions) without omitting or fabricating requirements.
- Checklist (3 items sampled uniformly per checklist to avoid revealing model identity via length; binary defect flags):
- D1 Not entailed: Requires something neither stated nor implied by the instruction.
- D2 Not verifiable: Cannot be judged objectively yes/no.
- D4 Incorrect: Factually wrong.
Results. (Δ = gemma-4-31B − gpt-oss-120b; 95% CIs via conversation-level bootstrap adjusted for annotator).
| Criterion | gpt-oss-120b | gemma-4-31B | Δ (95% CI) |
|---|---|---|---|
| I1 Clarity | 4.62 | 4.80 | +0.18 [-0.05; +0.43] |
| I2 Self-containment | 4.70 | 4.90 | +0.20 [-0.05; +0.50] |
| I3 Feasibility | 4.60 | 4.92 | +0.32 [+0.08; +0.60] |
| I4 Fidelity to the user | 4.34 | 4.74 | +0.40 [+0.09; +0.71] |
| Instruction mean | 4.57 | 4.84 | +0.28 [+0.06; +0.51] |
| D1 Not entailed | 13.3% | 5.3% | -8.0 pp [-15.0; -1.4] |
| D2 Not verifiable | 1.3% | 2.0% | +0.7 pp [-2.9; +3.8] |
| D4 Incorrect | 0.0% | 1.3% | +1.3 pp [+0.0; +4.4] |
| Any checklist defect | 14.7% | 8.7% | -6.0 pp [-14.5; +2.0] |
Takeaways.
- Cross-family prompt transferability (Weakness 1): Swapping to a different model family without prompt tuning preserves (and slightly improves) human-rated quality across criteria (+0.28 instruction mean; -8.0 pp non-entailed checklist items).
- Model-agnostic quality bar: Under both generators, incorrect and unverifiable checklist items remain ≤ 2% and instruction fidelity stays above 4.3/5.
Note that this experiment evaluates artifact synthesis quality rather than downstream Arena AI ranking correlation (which would require regenerating all 500 tasks and re-evaluating all 19 models, left for future work).
Dear Reviewer TB3f,
We would like to thank you again for the valuable feedback. The submission has been updated to address the points raised, including the new experiments. Please let us know if you have any remaining questions.
Best regards, The Authors.
We thank the reviewer for the constructive feedback and insightful comments on our work. Below, we address the specific points raised.
Weakness 2 - About the recency of the models
We evaluated 6 recent models (released in 2025–2026, 5 of them reasoning models): the benchmark keeps a high rank correlation with Arena AI and orders them close to the human-preference ranking (results below).
The selected model roster is relatively old because of the baseline benchmark generation method we compare our work to, which is Arena-Hard-Auto. Our baseline data relies on the results of models released by mid-2024 (Section 4.3; both references are frozen at 2024-07-31). To ensure a valid comparison, we pair evaluated models between the two methods. However, to test whether the generated benchmark also holds up when evaluating newer models, we chose 6 models released between mid-2025 and 2026, including 5 reasoning models, and evaluated them. Results are below.
Our benchmark preserves the Arena AI order of these models except for one (Muse Glimmer 30B, ranked 4th by Arena AI and 2nd by our benchmark), and when combining these 6 with the older 19 models, the ranking (gemma4-31b as judge) shows high correlation (Spearman 0.970 [0.874; 0.996]), agreement (86.8%) and separability (86.0%).
The results indicate that the newer models are slightly saturated on our benchmark, but this can be explained by the date of the conversations used to build it, which reflect older use cases. Despite this minor ceiling effect, the benchmark distinguishes these recent models from the 2024 roster and maintains high correlation in ranking. We will add these results to the revised manuscript. We agree with the reviewer that the comparison with Arena-Hard-Auto supports a “comparable” rather than a “better” claim, which is how we state it in Section 5.1 and Section 6. We also note that our primary contribution is the method of generating benchmarks and not the generated benchmark itself.
Question 1 - About cost-performance tradeoff
We believe that is the case, superior open-weight models are likely to be better at understanding user requirements and formulating instructions and checklists, so cheaper models could degrade the comparison with Arena-Hard-Auto. Our pipeline already uses the smaller gpt-oss-20b for the filtering stage (Section 4.1). However, the use of open-weight models itself for this task is already a novel contribution, and we leave a deeper exploration of this tradeoff for future work.
Weakness 1 - About the pipeline's robustness to the underlying model
We are running an experiment to evaluate this robustness. As soon as the results are available, we will post them on this thread.