- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework to optimize the mechanism parameters with limited budget and unknown behavior models. The framework consists of two steps. First, use LLMs to generate many candidate behavior models. Second, use Bayesian updates to update the posterior belief of the behavior models. The optimal mechanism parameters are determined given the posterior belief of the behavior models. The paper also considers two methods in second step: BUFS and MUFS. BUFS is more intuitive, while MUFS is more "end-to-end". Experiments show that BUFS and MUFS outperform other baselines in or out of sequential experimental design literature.
Critical strengths
- The proposed framework for sequential experimental design is quite interesting, though not very groundbreaking. I learn something new from the presentation. I would like to give a "weak acceptance" if all my concerns could be resolved.
Critical weaknesses
-
The paper claims that "using LLMs to generate behavior models" is part of its contribution. My concern is that this does not constitute a real contribution, because it is quite natural and standard. One could also generate behavior models through expert knowledge (e.g., enumerating the behavior models in existing literature). Furthermore, this paper does not validate whether the LLM-generated behavior models are better than other alternatives (e.g., enumerating the behavior models in existing literature), and the advantage of using LLMs is not very clear.
-
I'm unsure about the marginal contribution of this paper over existing literature. For this reason, I can not recommend acceptance at this stage. Specifically, in "Sequential Experimental Design" paragraph in "Related Works" section, the paper distinguishes itself from existing literature by arguing that existing literature require a ground truth behavior model, while this paper does not. The remedy is to use LLMs to generate several candidate behavior models. I do not see this contribution enough for acceptance (see my weaknesses 1).
-
I would like to see whether MUFS is a new approach for sequential experimental design. However, the paper does not give an affirmative answer to this question.
-
It's unclear whether the current baselines are appropriate. It seems to be that current baselines are weak. As the paper mentions that there have been many works in sequential experimental design, and regards proposing BUFS/MUFS as part of their contributions, it is important to compare BUFS/MUFS with some of them.
Questions for authors
Not really questions, but I do not know where to put them, and these points do not affect my rating of the paper.
- In equation (3), the paper introduces a metric to measure the quality of mechanism parameter , but do not evaluate it in experiments in the main text. I had expected to see this in the main text.
- The motivation for introducing as a surrogate of is confusing. The paper claims "Since the model-parameter space is high-dimensional, this results in a large number of optimization solves, making the overall computation particularly challenging", however, replacing with still requires a large number of optimization solves.
- The introduction of Proposition 1 is confusing as well. On one hand, the assumption seems quite strong to me; on other hands, the connection between Proposition 1 and the other parts of the paper is unclear. Furthermore, the statement "then the strict pairwise rankings of experiments induced by are preserved under " is not a rigorous statement, and one should avoid this.
- The experiment configurations are not clear enough in the main text. For example, the authors do not introduce the value distributions, the number of bidders, the behavior models generated by LLMs, the value of , the prior set to the behavior models and the parameters, that are important to uniquely define the experiments. In addition, in all experiments, and seem to refer the same thing, so it is confusing why the paper introduces two concepts in the main text.
- The "Problem Formulation" section is too abstract, I believe that a concrete example can bring a better understanding to the readers.
- The main text up to section 5 is a bit wordy, e.g., repeating the same meaning many times. I believe that this can be improved.
8 comments
Authors' Answers to Question 4, Question 5, and Question 6
Q4. The experiment configurations are not clear enough in the main text. For example, the authors do not introduce the value distributions, the number of bidders, the behavior models generated by LLMs, the value of K, , , the prior set to the behavior models and the parameters, that are important to uniquely define the experiments. In addition, in all experiments, e and z seem to refer the same thing, so it is confusing why the paper introduces two concepts in the main text.
We thank the reviewer for these questions!
Regarding configurations: We agree that the experimental configurations should be presented more clearly. We had moved the key settings from Appendix C to the main text. We set in Sections 5.1 and 5.2, while Appendix B.3 examines pool sizes ranging from to . For conciseness, below we summarize required experimental settings except LLM-generated behavioral models which are specified in Appendix C.4 in the revised paper.
| Config. | First-price Auction | Pay-per-bid Auction |
|---|---|---|
| Value distribution | i.i.d. . | Fixed |
| Number of bidders | ||
| Mechanism parameter space () | . | . |
| Design space () | . | . |
| Surrogate parameter space () | . | . |
| Behavioral model prior | Uniform: . | Uniform: . |
| Behavioral parameter prior | Independent uniform over and . | Independent uniform over and . |
Regarding and : Although both are expressed using the same auction parameters (i.e., the reserve price in first-price auctions and the bid fee and bid increment in pay-per-bid auctions), they serve different roles in our framework. Specifically, the experiment denotes the mechanism parameter tested in round to collect behavioral observations and update posterior beliefs. It is selected for the purpose of learning and information collection. In contrast, denotes the final mechanism parameter deployed after experimental rounds to maximize expected revenue. Therefore, the most informative experiment need not be the optimal final mechanism.
We have added relevant content to Section 5.1, Section 5.2, and Appendix D.2 in the revised paper.
Q5. The "Problem Formulation" section is too abstract, I believe that a concrete example can bring a better understanding to the readers.
We thank the reviewer for this question! We agree that a concrete example would help readers better understand the problem formulation. We have added the following example to Section 3 in the revised paper.
Consider a first-price auction where the designer aims to maximize revenue by choosing an appropriate reserve price. When the bidders' behavioral model is unknown, and the number of experimental rounds is limited, the designer sequentially conducts auctions with informative reserve prices , observes the resulting outcomes , and updates beliefs over the LLM-generated candidate behavioral models and their parameters. After experiments, the designer uses the updated beliefs to deploy the final reserve price that maximizes expected revenue.
Q6. The main text up to section 5 is a bit wordy, e.g., repeating the same meaning many times. I believe that this can be improved.
We thank the reviewer for this question! We agree that some parts of the main text are repetitive. We have refined relevant paragraphs in Section 3 and Section 5 to improve conciseness and clarity in the revised paper. For example, in Section 3 in the revised paper, we have replaced "Since the true behavioral model and its parameters are unknown, reducing uncertainty about them will help in reducing uncertainty about the optimal mechanism parameter" with "Since the true model-parameter pair is unknown, reducing uncertainty about it helps identify the optimal mechanism parameter".
Authors' Reply to Weakness 1
W1. The paper claims that "using LLMs to generate behavior models" is part of its contribution. My concern is that this does not constitute a real contribution, because it is quite natural and standard. One could also generate behavior models through expert knowledge (e.g., enumerating the behavior models in existing literature). Furthermore, this paper does not validate whether the LLM-generated behavior models are better than other alternatives (e.g., enumerating the behavior models in existing literature), and the advantage of using LLMs is not very clear.
We thank the reviewer for raising this concern! We agree that we should be more precise in positioning. Generating behavioral models with LLMs is not, by itself, a standalone technical contribution, and our key technical contribution is MUFS.
First, our conceptual contribution is to introduce the use of LLMs to generate candidate behavioral models for mechanism optimization under unknown strategic behavior and limited experimental budgets. To the best of our knowledge, we are the first to explore this idea in this problem setting and empirically demonstrate its feasibility. Specifically, LLMs offer a practical advantage: they can rapidly convert open-ended behavioral uncertainty into a diverse pool of plausible behavioral models without requiring extensive manual enumeration.
Second, our key technical contribution is MUFS. We develop sequential experimental design strategies, BUFS and MUFS, for mechanism optimization under unknown strategic behavior and limited experimental budgets. Given this candidate pool, BUFS focuses on distinguishing among them, while MUFS directly targets the optimal mechanism parameter. Please see our Reply to Weaknesses 2 & 3 for further details.
Third, generating candidate models with LLMs alone is insufficient for mechanism parameter optimization. This is further supported empirically: our methods and the sequential design baselines (i.e., Seq-XXX) use the same LLM-generated candidate model pool, yet achieve different results. Thus, the empirical performance improvements cannot be attributed solely to LLM-based model generation, but also reflect the effectiveness of our sequential experimental design strategies.
We have refined the positioning in Section 1 of the revised paper.
Authors' Answers to Question 1, Question 2, and Question 3
We thank the reviewer for this question! The metric in Eq. (3) cannot be evaluated exactly because the true agent behavioral model is unavailable. However, since all methods are evaluated against the same agents whose behavioral models are assumed to remain fixed throughout the experiments, the first term in Eq. (3) (i.e., the maximum achievable expected objective value) is constant across methods. Thus, comparing the second term (i.e., expected objective value under each method's optimized mechanism parameters) is sufficient to compare their relative gaps, with higher values indicating smaller gaps. In practice, we estimate this second term using empirical revenue averaged over 10 independent runs, each comprising 30 independent auctions.
We have added this discussion to Section 5 in the revised paper.
We thank the reviewer for this question! We acknowledge that the original explanation did not make this statement sufficiently clear. The key benefit of the surrogate space is not to reduce the number of model-parameter pairs considered, but to reduce the computational cost for each pair. Specifically, for each sampled model-parameter pair, computing $z^* $ requires solving an optimization problem over the full, fine-grained parameter space of $\mathcal Z $. In contrast, finding $\tilde z^* $ requires only evaluating and comparing the objective values over a small discrete space $\tilde{\mathcal Z} $. For example, in first-price auctions, we have $|\tilde{\mathcal Z}|=11 $. Thus, $\tilde z^* $ can be obtained by evaluating the revenue at these 11 points and selecting the one with the highest value.
We have added this discussion to Section 4.3 and Appendix D.2 in the revised paper.
We thank the reviewer for these questions about Proposition 1! We agree that the original presentation of Proposition 1 was not sufficiently clear or precise, and now we address your questions separately, as follows.
Regarding the assumptions: First, as discussed in Appendix A.3, we acknowledge that Proposition 1 is intended as theoretical motivation for the surrogate approximation rather than a strict implementation guarantee. Second, assumptions such as Lipschitz continuity are sufficient rather than necessary conditions. Moreover, to reduce the risk of substantially violating the assumptions, we can increase the density of representative points in the surrogate space. A denser set makes it more likely that z∗ lies close to at least one representative point, thereby improving the quality of the surrogate approximation.
Regarding its connection to MUFS: Proposition 1 justifies using the surrogate space for experiment selection in MUFS. MUFS ranks candidate experiments according to their criterion values, and selects the next experiment with the highest value. The original space Z and the surrogate space Z~ each induce a ranking of candidate experiments based on their respective criterion values. Since evaluation in the original space is expensive, MUFS uses the surrogate space instead. Therefore, it is important that the surrogate-space ranking agrees with the original-space ranking. Proposition 1 establishes precisely this property under the stated conditions, ensuring that the same optimal experiment is selected when it is unique.
Regarding the statement: We agree that the wording is imprecise. We have replaced it with the following statement: Under the stated conditions, for any two candidate experiments e1,e2∈E, S(e1)<S(e2) implies S~(e1)<S~(e2). Thus, the surrogate space preserves all strict pairwise rankings induced by the original space, ensuring that the same optimal experiment is selected when the optimum is unique.
We have refined the content in Section 4, Appendix A.2, and Appendix A.3 in the revised paper.