acceptodds
Withdrawn from ICLR 2027

Simulating Experiments for Method Testing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Field experiments (A/B tests) provide credible benchmarks for methods in societal systems, but their cost and latency slow method development. Synthetic evaluators, e.g., LLM-persona panels, offer a scalable alternative, raising the question of when they preserve the benchmark interface that adaptive methods optimize against. We show that replacing human evaluation with synthetic evaluation is indistinguishable at this interface from changing only the evaluation population if and only if (i) methods observe only aggregate outcomes (aggregate-only observation) and (ii) feedback depends on the submitted artifact rather than the method's identity or provenance (method-blind evaluation). LLM-persona panels instantiate this framework by sampling profiles and using conditioned LLMs to generate evaluations. We then separate validity from usefulness: an information-theoretic discriminability measure yields explicit bounds on the effective number of independent evaluation units needed to distinguish meaningfully different methods at a chosen resolution.

Primary area:
datasets and benchmarks
Keywords:
method benchmarking, field experiments, simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jF8kOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when synthetic evaluators, such as LLM-persona panels, can replace human field experiments as benchmarks for developing adaptive methods. Its main result (Lemma 3.2) shows that, from the method's point of view, swapping human for synthetic evaluation is equivalent to changing the evaluator population if and only if two conditions hold. First, the method sees only aggregate scores (AO). Second, the feedback does not depend on which method produced the artifact (MB). The paper also bounds how violations of (MB) accumulate over adaptive rounds. Finally, it proposes a discriminability measure κQ\kappa_Q that determines how many evaluations are needed to reliably compare methods, and applies it to two benchmarks (ad optimization and sales-call stopping).

Critical strengths
  1. The discriminability-based budget calculation is a reasonable sanity check. In the ad experiment, the required number of ratings varies widely across scenarios, suggesting that a single fixed persona-panel size may be inadequate.

  2. Proposition C.1 correctly shows that (AO) is necessary for the equivalence result.

Critical weaknesses

My main concern is that the paper formalizes evaluation-hygiene norms (not revealing method identity, exposing only aggregate scores) and then presents them as a theory of when synthetic experiments can substitute for human experiments. However, the paper does not resolve the core question that motivates it: when do conclusions drawn from comparing methods on QpersQ_{\text{pers}} continue to hold on QhumQ_{\text{hum}}? The paper gives no condition linking the two kernels, and neither the TextBO experiment nor the sales-simulator experiment includes any human data to check method rankings.

1. The introduction's central claim is unsupported.

  • The paper acknowledges that interface validity is distinct from fidelity to the human population (Introduction, Conclusion, Appendix F.3). Yet the introduction claims that improvement on a synthetic benchmark "may therefore provide sufficient evidence of the method's optimization capability needed to climb a human benchmark." This claim requires some relationship between the two kernels, and the paper does not provide one.
  • The paper's stated goal is a benchmark for comparing methods. At a minimum, this requires that comparisons between methods transfer from QpersQ_{\text{pers}} to QhumQ_{\text{hum}}. "Climbing ability" is not defined independently of the landscape (cf. the No Free Lunch theorems [3]). Because (AO)+(MB) admits arbitrary artifact-only kernels, it provides no guarantee of such transfer.

2. The experiments do not test this premise.

  • Neither application compares the same artifacts or methods under human evaluation.
  • In both settings, (MB) holds trivially by construction. Systematic preferences of LLM judges for particular stylistic features are functions of the artifact ww, so footnote 2 excludes them from (MB). The biases that actually threaten transfer therefore lie outside the framework.
  • The sales simulator's calibration diagnostics show that the two environments differ structurally from a method's point of view. The AUC for predicting early failure is only 0.600.60, versus 0.940.94 in the source study (Table 4). In Table 5, all three stopping rules yield lower fixed-budget sales than never stopping, whereas the premise of the source study is precisely that stopping early increases sales by reallocating time.

3. Limited novelty.

  • (AO) and (MB) are essentially existing evaluation norms. The paper itself lists Blum & Hardt [1] in Appendix A as the closest precursor to (AO), and (MB) is already standard practice in the anonymous pairwise comparisons of Chatbot Arena [2].
  • The theoretical results beyond the core characterization (Theorem 3.3, Proposition 3.4) are correct but fairly elementary.

References

[1] A. Blum and M. Hardt. The Ladder: A Reliable Leaderboard for Machine Learning Competitions. In Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37, pp. 1006–1014, 2015.

[2] W.-L. Chiang, L. Zheng, Y. Sheng, et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235, pp. 8359–8388, 2024.

[3] D. H. Wolpert and W. G. Macready. No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation, 1(1):67–82, 1997.

Questions for authors
  1. Beyond (AO)+(MB), what conditions must QhumQ_{\text{hum}} and QpersQ_{\text{pers}} satisfy to guarantee that method rankings on the synthetic benchmark predict method rankings on the human benchmark? For example, both kernels might belong to a class of landscapes on which comparisons between methods remain stable. If this is beyond the scope of the paper, how should the introduction's claim about transferable "climbing" ability be understood?

  2. Is any human comparison data available for either application? Even a small-scale study would materially change my assessment. For example, do human raters also prefer the ad images produced by the prompt improvements that TextBO selects?

  3. Regarding clustering and the variance convention:

    • In the sales benchmark, sellers and buyers form a clustered structure (each of the 80 sellers appears in many calls, and each buyer appears once per scenario). Section 5 states that all sample sizes refer to effective independent units (Appendix F.1). Were the corresponding ICCs or effective sample sizes actually estimated?
    • Is κ^Q\hat{\kappa}_Q in Table 6 computed from the variance of paired differences, as described in Appendix E, or from the independent-sample variances of the two policies?
  4. Appendix E notes that κQ\kappa_Q could be computed under other choices of ν\nu, such as threshold perturbations or neighboring deadline rules. How sensitive are κQ\kappa_Q and the required budget to the choice of ν\nu?

Reviewer_5RSrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper contributes a series of theoretical results around the use of synthetic panels powered by LLM in place of panels composed of human experts, when the purpose of the panel is not to represent human preferences as such but to provide feedback to an adaptive method optimising towards the corresponding judgments. The theoretical results include characterising necessary and sufficient conditions for the swap between human and synthetic panel to be equivalent to a change in the composition of the human panel, and bounds based on the violation of such conditions. They also include some results about the usefulness of synthetic panels, in terms of their ability to discriminate between method outputs (and therefore methods) depending on the effective size of the panel and the signal to noise ratio of the judgments.

Critical strengths
  • Since synthetic panels are increasingly used, contributions that clarify what can be gleaned from them are welcome.
  • The formalization of concepts and properties to talk about synthetic panels is potentially impactful.
Critical weaknesses
  • The paper body is far from self-contained. Since all proofs, experimental validation, and all related work are relegated to the appendix, it is not possible to formulate an opinion about the soundness of results or their relative novelty w.r.t. prior work by reading the body of the paper (which is what readers are expected to be able to do).

  • It is not clear from the paper if the core results are not vacuous. What does Literal Panel Change guarantee in practice? Is the distribution over panelists allowed to change arbitrarily? If so, what guarantees are there that the equivalence does not hold only for degenerate choices of the replacement panel? Also, some assumptions are made (e.g. homoscedastic Gaussian reduced-form model) for some results without being neither explained nor justified.

  • In several cases, notation and terminology are used without or before being properly introduced (e.g. ‘micro-instrument’, dTVd_{TV}, LB\cal{L}_B, ...).

  • Little attention has been placed in connecting formal properties with practical considerations, so it is not clear how significant results are, if they hold.

Questions for authors
  • What is the practical bearing of the results in Section 3.2?
Reviewer_4btLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks when an LLM-persona panel can replace human A/B tests for evaluating adaptive methods, i.e. optimisers that learn from feedback.

Critical strengths
  1. The question is relevant, and AO/MB is a useful checklist. Persona panels are being adopted quickly, often with little thought about validity. Separating "the simulator reproduces humans" from "the simulator is a valid testbed for an optimiser" is a useful distinction. Two pieces of advice are sound practice: do not show the optimiser individual persona outputs, and do not let the evaluator see which method produced an artifact.

  2. The practical power warning is useful.

Critical weaknesses
  1. The identification result holds by construction. Lemma B.2 builds the "panel change" with a dummy two-point panel {0, 1}: humans are δ0, personas are δ1, and the instrument simply returns Q or Q′. Weired,

  2. There is no human data, and the estimates are fragile. The thesis concerns substituting synthetic evaluation for human evaluation, yet neither case study includes a human comparison.

  3. The simulator is poorly calibrated to the human data it is based on: AUC 0.60 versus 0.94, and successful calls of 162.7s versus 630s.

  4. Could the authors answer: a. Given the dummy panel in Lemma B.2, what does "panel change" add beyond MB? Is there a version in which the panel has real structure? b. Can you give confidence intervals for κ̂ and estimates of the effective sample size? c. A small human comparison for either case study would ground the framework.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.