- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes an evidence-acquisition method that combines predicted utility-covariance reduction with the Hessian of an entropy-smoothed allocation value. Its aim is to acquire information that distinguishes competitive feasible allocations, rather than merely reducing prediction uncertainty. This is an interesting idea and a sensible objective to me, I also find the allocation-contrast interpretation is useful. The authors evaluated their approach using the publicly available dataset.
Critical strengths
I think the problem studied by the authors is important and I appreciated the focus on collecting information to make better decisions rather than simply reducing uncertainty.
Critical weaknesses
Despite solving an interesting problem I still have a few concerns. First, I find the clarity of the paper can be substantially improved. In particular, Section 2 compresses the entire sequential decision problem into one paragraph. Evidence blocks, masks, legal queries, and policy/source descriptions appear without a worked example connecting these objects. More importantly, it does not explicitly specify the joint probabilistic model of utilities and observations, the source of randomness in the expected objective, or the admissible adaptive query policy. Some definitions arrive later, but the reader must reconstruct the setup across multiple sections. The paper should first give a self-contained formulation and perhaps a concrete example. Additionally, the reproducibility of the work is also concerning. For instance, Appendix C.1 does not specify the training loss for the utility mean and covariance, as well as the reproducible model architecture and training configuration. It also says that temperature is computed from predicted utilities without specifying how exactly it is computed. The rank-four covariance parameterization also needs justification and sensitivity analysis. Without these details the evaluated system is not reproducible. The evaluation section can be significantly improved. For instance the baselines are not properly explained, for example, “Uncertainty” which is the strongest external comparator is particularly underspecified. More importantly, it is not clear whether Table 2 compares methods with the same predictors and, therefore, I am not sure if the reported results cleanly attribute improvements to MOVE’s acquisition score. I would also suggest to the authors to better connect their papers to the existing work. For example, the discussion omits particularly relevant work on bandit-based approaches such as Chen et. al. “Combinatorial Pure Exploration of Multi-Armed Bandits” (Neurips, 2014) or Fiez et. al. “Sequential Experimental Design for Transductive Linear Bandits” (Neurips 2019)
Overall, the paper contains a potentially useful acquisition heuristic, but needs much more clarity in explanations, positioning in the related work and having more reproducible implementation details.
Questions for authors
(see all my questions above)
3 comments
Response to Reviewer biad — Part 2 (Q5–Q8)
Q5: “The rank-four covariance parameterization also needs justification and sensitivity analysis.”
Answer: This is a useful sensitivity check. We followed your suggestion by varying the number of covariance factors.
Table R1. Allocation utility with different numbers of covariance factors.
| Factor rank | Utility ratio ↑ | 95% CI |
|---|---|---|
| 2 | 0.868 | [0.835, 0.899] |
| 4 | 0.878 | [0.852, 0.902] |
| 8 | 0.853 | [0.824, 0.882] |
Four factors achieve the highest mean utility among the tested settings, supporting the default choice. The 95% CIs are obtained by bootstrapping the 32 test county subsets, keeping each subset’s three objectives together and each trained model fixed (Appendix C.3).
Q6: “The evaluation section can be significantly improved. For instance the baselines are not properly explained, for example, “Uncertainty” which is the strongest external comparator is particularly underspecified.”
Answer: Good point. Here is the exact procedure. Uncertainty uses its own ridge-regression utility predictor and a fixed variance-based score for each evidence block. The block score is the mean training-set variance across its available features. At each step, every unrevealed candidate–block pair receives its block’s score, and the highest-scoring legal query is selected with deterministic tie-breaking. Revealed evidence is added to the observed state; after the query budget is exhausted, the predictor estimates utilities and the shared exact solver returns the allocation. We will add these implementation details to Appendix C.2.
Q7: “More importantly, it is not clear whether Table 2 compares methods with the same predictors and, therefore, I am not sure if the reported results cleanly attribute improvements to MOVE’s acquisition score.”
Answer: We agree that the contribution of the acquisition score needs a controlled test. We therefore ran additional ablations using the same trained predictor with all its parameters fixed. The initial observations, four-query budget and final solver are also shared.
Table R2. Ablations with the predictor parameters fixed.
| Method or variant | Utility ratio ↑ | Jaccard ↑ |
|---|---|---|
| MOVE | 0.878 | 0.486 |
| w/o allocation sensitivity | 0.835 | 0.441 |
| w/o covariance contraction | 0.830 | 0.449 |
With the same frozen predictor, MOVE outperforms both acquisition-score ablations on utility and Jaccard, supporting the contribution of its query-selection rule.
Q8: “I would also suggest to the authors to better connect their papers to the existing work. For example, the discussion omits particularly relevant work on bandit-based approaches such as Chen et. al. “Combinatorial Pure Exploration of Multi-Armed Bandits” (Neurips, 2014) or Fiez et. al. “Sequential Experimental Design for Transductive Linear Bandits” (Neurips 2019)” “Overall, the paper contains a potentially useful acquisition heuristic, but needs much more clarity in explanations, positioning in the related work and having more reproducible implementation details.”
Answer: These are relevant work, and we appreciate the pointers. Both papers study how to acquire information to identify the best decision, but their observation models differ from ours. [Chen et al. (2014)] collect repeated reward samples from individual items. Our queries reveal a missing feature block once, and utility labels cannot be queried (Section 2; Appendix B.3). [Fiez et al. (2019)] assume a specified linear relationship between measurements and decision values. MOVE uses a learned model to predict how revealing a feature block changes candidate utilities. We will cite both papers and explain these connections and differences in Section 5.
Thank you for responding to my review. I will maintain my current score for now.
Response to Reviewer biad — Part 1 (Q1–Q4)
We are glad you found the focus on better decisions valuable. Below, we clarify the method, report the requested sensitivity analysis, and provide additional ablations with the predictor parameters fixed.
Q1: “First, I find the clarity of the paper can be substantially improved. In particular, Section 2 compresses the entire sequential decision problem into one paragraph. Evidence blocks, masks, legal queries, and policy/source descriptions appear without a worked example connecting these objects.” “Some definitions arrive later, but the reader must reconstruct the setup across multiple sections. The paper should first give a self-contained formulation and perhaps a concrete example.”
Answer: A worked example would help here. Consider selecting four of 15 counties within a budget, including at least two high-vulnerability counties. The policy description specifies the equity objective. Suppose two valid selections differ only in whether they include County A or County B. County A’s surveillance block is missing; its source description says it contains recent case counts and trends. The mask marks this block as unobserved, so querying it is legal. Revealing the values updates the utility predictions and may change which selection is preferred. After four queries, the solver returns the valid selection with the highest predicted total utility—the slate. We will place this example before the formal definitions in Section 2 (Appendix B.3, Table 7; Appendix B.4, Eq. (26)).
Q2: “More importantly, it does not explicitly specify the joint probabilistic model of utilities and observations, the source of randomness in the expected objective, or the admissible adaptive query policy.”
Answer: We agree that these three elements should be stated together. We will make them explicit in Section 2:
Q3: “Additionally, the reproducibility of the work is also concerning. For instance, Appendix C.1 does not specify the training loss for the utility mean and covariance, as well as the reproducible model architecture and training configuration.” “Without these details the evaluated system is not reproducible.”
Answer: We appreciate the request for a precise training specification. The utility mean, covariance and missing-evidence predictions are trained jointly from randomly masked inputs; the text encoder remains frozen.
Loss. We minimize the joint negative log-likelihood of the utility vector and hidden measurements, conditioned on the observed blocks and allocation objective:
=-\mathbb E_{t,O}\log q_\theta \left(u_t^\star,x_{t,H}\mid x_{t,O},p_t\right).$$ Here $O$ is the randomly sampled set of observed blocks, $H$ contains the hidden blocks, and $p_t$ specifies the allocation objective. **Architecture.** A frozen **e5-mistral-7b-instruct** encoder embeds objective and source descriptions. Trainable projections combine these embeddings with observed values, masks, missing-block embeddings and public metadata. The output heads predict hidden measurements, utility means $\hat\mu_\theta\in\mathbb R^{15}$ and covariance $C_\theta=D_\theta+L_\theta L_\theta^\top$, where $L_\theta\in\mathbb R^{15\times4}$ and $D_{\theta,ii}=\operatorname{softplus}(\eta_i)+10^{-6}$ (Section 3.1; Appendix C.1). **Training.** We use batches of eight episodes for up to 20 epochs and select the model on the development set. We will add the implemented likelihood factorization, layer dimensions and optimizer settings to Appendix C.1. **Q4:** “It also says that temperature is computed from predicted utilities without specifying how exactly it is computed.” **Answer:** You are right that the numerical rule should be explicit. **We set the temperature using the standard deviation of the predicted utilities:** $$\tau=\operatorname{clip} \left(0.35 \operatorname{sd}(\hat\mu), 0.035, 0.12\right).$$ The standard deviation is computed across candidates. We recompute $\tau$ after each real query and keep it fixed while scoring the possible next queries (Section 3.2; Appendix C.1). We will add the numerical rule to Appendix C.1.