- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an approach to scaling the Delphi method with the use of LLMs. The Delphi method is the established methodology for high-stakes decisions when they cannot be answered via direct measurement. This entails experts providing probability estimates with written rationales, which are iteratively revised until convergence. This process is both expensive and time-consuming, The proposed framework swaps out experts for LLMs because they can provide probably estimates and be queried repeatedly for modified conditions and/or assumptions.
The provided results suggest that Scalable Delphi is a valid framework, with LLM probability estimates exhibiting similar trends to the experts on the included benchmarks. There are obviously differences but the LLMs appear to agree within the bounds of the differences across experts.
Critical strengths
Firstly, the selected problem seems to be an ideal setting for the use of LLMs. The primary challenges with Delphi are the cost, both in resources and time, of engaging with experts over multiple rounds until convergence. This presents a fertile ground for the use of LLMs due to their relative cost-effectiveness and speed. Additionally, the ability to repeatedly query them as conditions shift offer an additional advantage over the use of experts in Delphi. The presented results substantiate the case for Scalable Delphi being a viable alternative to Delphi; further testing in real settings is probably necessary but the included benchmarks make a fairly compelling case.
I also greatly appreciate the authors' writing. The manuscript is clearly structured, with notation clearly explained and used to clarify meaning. Even as a reader relatively new to the domain of expert elicitation, i was able to follow the reasoning and explanations. The experiments and benchmarks have also been structured to paint a clear picture behind the design choices made and validity of the proposed method.
Including multiple benchmarks was a nice touch too as it exemplifies that Scalable Delphi was not designed with a specific domain in mind. Delinating the limitations of the alternate benchmark was useful too. This further strengthened the case for Scalable Delphi with the appropriate caveats for readers and practitioners when viewing the results,
Critical weaknesses
The manuscript would benefit from a more detailed discussion of the related works. At present, this has been included in Section 6 and reads almost like an afterthought. However, this discussion is important to position this work relative to existing research.
I have also included some pointers and questions in the next section.
Questions for authors
- On line 76, the authors state that "the relevant quantities can be elicited again". While I understand how this is useful, especially relative to experts, does this not introduce the risk that repeated queries may lead to a "shift" in the LLM outputs? This could be problematic for how well-calibrated the model probability estimates are.
- The frontier models used are relatively old (GPT 5.1 AND Claude Opus 4.1). Why were these chosen? And are there any reasons to expect Scalable Delphi to not work with newer models?
- It would be useful to define ECE in the main text, instead of the appendix because it is one of the key metrics. Additionally, contextualizing what good values for MAE and ECE in Table 1 would be useful. Currently, it is unclear whether a MAE of 4.5 is any good.
2 comments
Thanks for onboarding reviewer comments
I thank the reviewers for onboarding the concerns raised by the reviewers. The clearer positioning should certainly help distinguish this work relative to existing research in this field.
I maintain that this manuscript is worthy of acceptance and specifically, that the revisions have gone a long way in addressing the concerns raised by Reviewer WAXg.
Thank you for the careful review and for the follow-up comment on positioning. We have reworked the related work accordingly (W1), report run-to-run variation for re-elicitation (Q1), and moved the ECE definition into the main text with context for the reported values (Q3).
W1. Related work
We agree that the distinction from prior LLM-based Delphi work deserves more prominence than a closing related-work section gives it. In the revision, related work follows the introduction, is expanded, and states the distinction explicitly (see the general response).
Q1. Does re-eliciting risk shifting the outputs, and with it their calibration?
Re-elicitation is a fresh run, so the only source of change is a deliberately changed input. Each rerun starts from an empty context, without memory of earlier answers (S3.2; App. E). Keeping all other parameters fixed (model, settings, questions) therefore allows running a fresh study without contamination from prior runs. The protocol is meant to move estimates when the evidence changes and leave them in place otherwise: withholding agent and benchmark identities degrades correlation to noise, and supplying them restores it (S5.3). Variation between runs with the same evidence is small: rerunning an identical Cybench configuration gives r = 0.967 and 0.983 (MAE 4.2 and 3.8 pp) for GLM-5, and r = 0.738 and 0.737 (MAE 10.8 and 11.0 pp) for Llama-4 Maverick. Unlike a reconvened human panel, a fresh run also does not anchor on earlier answers (S5.4, S6).
Q2. Why GPT-5.1 and Opus 4.1, and should newer models work?
They were chosen for their knowledge cutoffs (September 2024 and January 2025; S5.1), which predate both human studies (Murray et al., March 2025; Barrett et al., December 2025) and two of the three benchmarks (BountyBench, CyberGym). Validating against these requires models that cannot have seen them. Newer models are likely to have both the studies and the leaderboards in their training data. Nothing in the method ties it to these two models, and we have no reason to expect newer models to perform any worse. GLM-5 and Kimi K2.5, released in 2026, perform on par with them (Table 1), and the only model that falls behind on Cybench is the one without extended reasoning (Llama-4 Maverick, S5.2). For a new model, the proxy tasks can be rerun on benchmarks released after its cutoff before it is used.
The contamination risk for newer models is real. In an additional check, we tested two state-of-the-art models, GPT-6.1-Sol and Claude Opus 5.5. They reach r = 0.96–0.99 in the full-information setting, but Opus 5.5’s reasoning cites recalled leaderboard figures. The paper’s models show no such recall (App. G).
Q3. ECE in the main text; what are good values for MAE and ECE?
We have moved the ECE definition into S5.1 and added the following context to S5.2. Thank you for the suggestion!