acceptodds
Under review as a conference paper at ICLR 2027

Active Query Synthesis for Learning from Preferences

Abstract

Most response models for pairwise preference comparisons assume all feedback is equally reliable, overlooking that comparisons between near identical or highly dissimilar options are often ambiguous. We introduce a *confidence aware response model*, where response reliability depends on the similarity of the compared options. A user study we conducted shows it models human choices better than standard models. Separately, active learning reduces the cost of estimating preferences, but pool-based methods evaluate the available candidate queries each round. Building on our response model, we propose *Info-Synth*, an active query synthesis framework. It selects informative queries by maximizing mutual information in closed form, up to a one-dimensional search, avoiding pool-based search entirely. For fixed datasets, we introduce two approximation strategies, *Pair M-dist* and *Pair Opt-dist*, to select the closest available pairs of points. We evaluate our framework on synthetic preference learning, a constrained text-summary dataset, and continuous-space controller gain tuning for a simulated robot. In each case, it matches or exceeds the sample efficiency of pool-based active learning while reducing the computational cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
active learning, preference learning, query synthesis, mutual information, hyperparameter tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8MzjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies active preference learning from pairwise comparisons under the ideal-point model: items are points in Rd\mathbb{R}^d, and for a query (p,q)(p,q), the user prefers the item closer to a hidden ideal point ww. The authors propose a confidence-aware response model in which the noise on each squared distance scales with that distance: P(Y=1∣w)=Φ ⁣(∥w−q∥2−∥w−p∥2σ0∥w−q∥4+∥w−p∥4)P(Y=1\mid w)=\Phi\!\left(\frac{\|w-q\|^2-\|w-p\|^2}{\sigma_0\sqrt{\|w-q\|^4+\|w-p\|^4}}\right).

Under a Gaussian posterior W∼N(μ,Σ)W\sim\mathcal{N}(\mu,\Sigma), Info-Synth maximizes I(W;Y∣p,q)I(W;Y\mid p,q) with the query p⋆=μ+r⋆v1p^\star=\mu+r^\star v_1, q⋆=μ−r⋆v1q^\star=\mu-r^\star v_1, where v1v_1 is the principal eigenvector of Σ\Sigma and r⋆r^\star comes from a 1-D minimization of expected conditional entropy. For fixed datasets, Pair M-dist (a Hessian-based Mahalanobis distance) and Pair Opt-dist (a heuristic built from the optimality conditions) shortlist candidate pairs and then pick the one with the highest exact MI. Experiments cover a 20-participant color study, synthetic data (2-10 dimensions), the Reddit TL;DR dataset (128-dimensional embeddings), and simulated gain tuning of a unicycle controller. The authors report sample efficiency comparable to or better than that of pool-based selection at a lower per-query cost.

Critical strengths

The query rule is interpretable and cheap: it centers the pair at μ\mu, aligns it with v1v_1, and sets the spacing via a 1-D search, which, in the continuous setting, costs O(Sd2)O(Sd^2) and requires no candidate scoring. In the discrete setting, scoring pairs jointly, rather than taking independent nearest neighbors of p⋆p^\star and q⋆q^\star, preserves the query's informativeness. In Figs 5 and 13, NN Approx and kk-NN Approx saturate while Pair M-dist tracks Active Discrete, so joint scoring recovers the performance that independent neighbor selection loses.

Critical weaknesses

The theory is weaker than claimed. With A=∥w−q∥2A=\|w-q\|^2 and B=∥w−p∥2B=\|w-p\|^2, the model's argument f=A−Bσ0A2+B2f=\frac{A-B}{\sigma_0\sqrt{A^2+B^2}} depends only on A/BA/B, so ∣f∣≤1/σ0|f|\le 1/\sigma_0; if p=wp=w and qq is far away, the model assigns its maximal confidence Φ(1/σ0)\Phi(1/\sigma_0), contrary to the motivation in Fig 1(c). Moderate spacing arises only from the posterior-level optimization, and since v1v_1 comes from the surrogate max⁡ E∣f(W)∣\max\,\mathbb{E}|f(W)| (App C.4) while r∗r^* comes from expected entropy, global optimality for the MI objective is not established. The experiments also fall short of the claims. In the user study (Fig 3), each condition changes both the response model and the query points, and σ0\sigma_0 was fit to the same participants, so Fig 3(b)-(c) cannot isolate which model better explains human choices; Fig 3(d) reports 1.61 vs 1.53 mean confidence from 20 participants without error bars or significance tests. The runtime gain (Alg 2, Fig 6) is small, 0.21-0.49 s vs 0.68-0.74 s per query, since the filter step still scores every candidate pair, and the fastest setting (γ=0.2\gamma=0.2) needs more queries. Under model mismatch (Figs 16-20), Info-Synth plateaus at a higher error than Active Discrete and loses its edge over Gauss Search Synthesis, so the gains depend on users following the assumed model, which Fig 3 does not establish.

Questions for authors
  1. Can you compare the response models on a common held-out set of human comparisons, with σ0\sigma_0 calibrated on separate data?
  2. How does query-selection time scale with NN and dd, given that the filter scores all pairs, and when is the saving meaningful end-to-end?
  3. Can v1v_1 and r∗r^* be shown to globally maximize the original MI objective? If not, under what assumptions is the result approximate?
1 comment

For the optimality concerns (Q3), please see our common response and the revised Sec. 4 and App. C in the paper. Questions about the user study calibration and computational details are addressed in the common response.

1. Distance-dependent confidence and Figure 1

Regarding the observation that ∣f∣≤1/σ0|f|\leq 1/\sigma_0, with maximal confidence when p=w\mathbf{p}=\mathbf{w} and q\mathbf{q} is far away: this is correct for a fixed user, and it is exactly why the two entropy terms cannot be separated. Such a query is answered confidently by every plausible user, so its outcome is predictable and it carries no information; the mutual information penalizes this through H(Y)H(Y). The preference for moderate separations is indeed a property of the posterior-level objective, and we now say so in the paper.

Figure 1 compares different separations for equiprobable queries, for which I(W;Y)=1−H(Y∣W)I(W;Y)=1-H(Y\mid W). Within the illustrated family, MI is lower at both separation extremes and maximized at an intermediate radius that minimizes response uncertainty averaged over the posterior. We have updated this detail in the paper.

2. Human responses and model mismatch

Figure 3 evaluates actual human judgments, providing evidence for the combined procedure in a real task, while Figures 19-23 test response mechanisms that differ from our assumed model. The results are therefore compatible. Info-Synth and Gauss Search Synthesis have similar query geometry, with separation being a principal distinction. Under the normalized mechanism, separation does not affect response probabilities at a fixed hyperplane; under the plain mechanism, larger gaps can improve reliability. The finite optimal-spacing advantage targeted by Info-Synth is therefore absent, which is consistent with the similar performance in Figures 19-20. Random gaps may occasionally favor Gauss Search; Info-Synth also updates its gap as the posterior changes. Active Discrete achieves lower errors in the reported mismatch settings at greater computational cost.

3. Reddit experiments: dimensionality and runtime

Dimensionality. All Reddit experiments now use the original 512-dimensional embeddings without PCA. The 128-dimensional setting has been removed.

Runtime. We replaced average per-query time with total query-selection time over all 80 queries. With the revised Pair Opt-dist score (Eq. 9), the smallest shortlist (γ=0.2\gamma=0.2) no longer needs more queries. Its accuracy matches Active Discrete's at every query count in 17 of 20 user–noise settings, and the largest deviation is 1.4 percentage points. Across 10 users, the median total query time is 36.8% of Active Discrete's (Figure 29). For example, for User 1 it is 124.4 s versus 339.1 s at σ0=0.1\sigma_0=0.1, and 86.7 s versus 263.1 s at σ0=0.3\sigma_0=0.3 (Figure 6). Because both methods need the same number of queries to reach a given accuracy, this saving also applies end to end, in time to a target accuracy.

Reviewer_G9AsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies active learning of a user's ideal point w in an embedding space from pairwise comparisons. It proposes a "confidence-aware" Thurstone-style response model (Eq. 2). The noise in each perceived squared distance scales with that distance, so the choice probability depends on the ratio of the two items' distances to w. For a fixed midpoint and user location, both very small and very large separations give unreliable answers.

With a Gaussian posterior N(µ, Σ), Info-Synth builds the next query directly:

the midpoint is placed at µ (the condition for equiprobable answers); the direction is the top eigenvector of Σ; the half-separation r* is found by a 1-D search that minimizes expected conditional entropy. For fixed datasets, two filter-and-refine heuristics are proposed. Pair M-dist uses a Mahalanobis distance from the MI Hessian, and Pair Opt-dist uses deviation from the optimality conditions. Both keep the candidate pairs closest to the built query and then choose among them by exact MI.

The evaluation has four parts:

a 20-participant color-matching user study; synthetic 2-D, 4-D and 10-D experiments; per-user preference learning on Reddit TL;DR embeddings (128-D and 512-D, three users); simulated tuning of controller gains.

Critical strengths

It tackles a real modeling issue. Standard BT/plain models favor far-apart queries, which may elicit unreliable answers. Building a query is cheap: one eigendecomposition plus a 1-D search. b = µ is proven sufficient for equiprobable answers (App. C.3), and the Hessian algebra checks out. The evaluation is broad and candid. It covers continuous and pool-based settings, includes a real user study and real preference data, and its mismatch tests show Pair M-dist keeping its relative performance.

Critical weaknesses

Gaps in the derivation: At lines 724–727, aᵀµ = τ does not imply b = µ. The two MI terms are optimized separately, with no proof that this gives the joint maximum. Minimizing entropy is replaced by maximizing E|f|, which is a surrogate. Prop. 1 drops the dependence on ‖S‖² and mixes two objectives. Pair M-dist assumes a stationary maximum. Algorithm 1 searches over a = r·v₁ but outputs a = 4r*·v₁. Confounded user study: Accuracy is measured on each model's own past choices. Each condition selects its own queries. The noise values were fit on the participants' responses. The study adds an ε noise floor, so it doesn't test the unmodified Eq. 2. No per-condition counts or significance tests are reported, and the shaded bands are undefined. Abstract overgeneralizes. Under a mismatched user model, Info-Synth only ties Gauss Search and loses to Active Discrete (lines 1511–1543). Efficiency is not attributed. There is no ablation that swaps the filter for a random subset, no comparison with Canal et al. (2019)'s MCMV/EPMV strategies, and the direction term in Pair Opt-dist hurts in 512-D (Fig. 26). Reproducibility gaps. β, η, k, S, the MCMC settings, run counts, and the gain-tuning details are missing, and the Reddit evaluation covers only three users.

Questions for authors

Can you report held-out log-likelihood for all response models fit to the same responses, plus per-condition n and significance tests? Were σ0 = 0.3 and 2.0 fit on the data shown in Fig. 3? What is ε? Can you prove that the separately optimized (a*, b*) is the joint MI maximizer? How does Prop. 1 handle ‖S‖², and how should the factor-of-4 mismatch in Algorithm 1 be read? Is −H(z*) positive semidefinite in practice, and what is done when it is not? What happens with a random-subset filter of the same size followed by the same MI refinement? How does the method compare with MCMV/EPMV and with gradient-based MI maximization? How were the Reddit users sampled? Was the projection head trained on data overlapping the held-out labels? What are the missing hyperparameters?

1 comment

The optimality concerns are addressed in common response and the revised Sec. 4 and App. C in the paper.

Concerns about the user study and gradient based MI maximization are in the common response.

1. Connection with MCMV/EPMV

EPMV and MCMV share the design rationale behind Pair Opt-dist: favor balanced comparisons along directions of high posterior uncertainty (Canal et al., 2019, Eqs. (8)–(9)). Pair Opt-dist adapts this rationale to our confidence-aware response model by penalizing deviations from the posterior center, a principal uncertainty direction, and the model-derived query separation. Their likelihood uses a pair-dependent noise parameter, whereas our noise scale also depends on the unknown user’s distances to both items, changing the appropriate query geometry and scoring rule.

2. Reddit experiments: Pair Opt-dist ablation, user selection, and reproducibility

Direction term in 512 dimensions. We revised the Pair Opt-dist score (Eq. 9) to separate query magnitude from direction, and we added single-term ablations (Figures 30–31; details in the common response). With the revised score, Pair Opt-dist matches Active Discrete in 512 dimensions. The magnitude term, which favors queries at an informative separation, has a Spearman correlation with MI close to −1-1 and by itself matches Active Discrete. On its own, the direction term is still weak. Candidate query directions are nearly orthogonal to the leading posterior eigenvector for almost all queries (Figure 31), so the term barely varies across candidates (Figure 30d) and cannot discriminate between them. In the combined score it does not hurt.

More users and full dimensionality. Results now cover 10 users (Figures 6, 26–29) in the original 512-dimensional embedding space, without PCA. We now report total query time over all 80 queries. Across all users and both noise levels, Pair Opt-dist (γ=0.2\gamma=0.2) shows no loss in accuracy compared with Active Discrete (largest deviation 1.4 percentage points), at a median of 36.8% of its total query time.

How users were selected. Users were not sampled uniformly at random. We chose the users with the most comparisons (475–1,874 each; User 1 has 1,874), so that each has a sufficiently large query pool and held-out set. App. F.3 now states this explicitly.

Run counts and uncertainty. Each experiment uses 5 trials, each with a different random 70/30 split into query pool and held-out evaluation set. Shaded bands in Figures 6 and 26–28 show the standard error across trials.

Projection head. We use the frozen projection head from prior work (Chen et al., 2025), which was trained on the TL;DR comparisons and so may have seen comparisons in our held-out sets. This could make absolute accuracies optimistic. However, every method (Active Discrete, Random Discrete, and Pair Opt-dist) uses exactly the same embeddings, so the comparison between query-selection strategies remains like-for-like.

Hyperparameters. In the Reddit experiments, η2=η3=1\eta_2=\eta_3=1, γ=0.2\gamma=0.2, and σ0∈{0.1,0.3}\sigma_0\in\{0.1,0.3\} (both reported), with 80 queries and 5 trials. None of these were tuned on held-out data.

Reviewer_aURZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies active preference learning through pairwise comparisons. It introduces a confidence-aware response model that incorporates distance-dependent response uncertainty, and proposes Info-Synth, a query synthesis method that seeks to maximize mutual information by reducing query optimization to a one-dimensional search. For constrained datasets, the authors develop two filter-and-refine approximation strategies, Pair M-dist and Pair Opt-dist. Experiments on synthetic datasets, a human color estimation study, Reddit TL;DR preference data, and simulated robot controller tuning demonstrate encouraging sample efficiency and computational savings.

Critical strengths
  1. Relevant problem and intuitive motivation. Incorporating response reliability into active preference learning is meaningful, particularly when human feedback is expensive or noisy.
  2. Practical and computationally motivated framework. The combination of continuous query synthesis and discrete filter-and-refine approximation provides a useful approach to reducing expensive mutual-information evaluations.
  3. Diverse experimental evaluation. The paper includes a human color estimation study, synthetic experiments under different noise levels, held-out preference prediction on Reddit TL;DR, and simulated controller tuning.
  4. Useful additional analyses. The appendices include model-mismatch experiments, ablations of the approximation heuristics, evaluations across additional users and embedding dimensions, and wall-clock timing comparisons.
Critical weaknesses
  1. Insufficient justification of the claimed global optimality.

    The derivation in Section 4 and Appendix C does not appear to fully establish that the synthesized query globally maximizes mutual information. In particular, maximizing the marginal response entropy by setting the query midpoint to the posterior mean does not by itself guarantee maximization of the full mutual-information objective.

    Moreover, Appendix C.2 derives b=μb=\mu from a⊤μ−τ=0a^\top\mu-\tau=0. Since τ=a⊤b\tau=a^\top b, this condition only implies a⊤(μ−b)=0a^\top(\mu-b)=0, rather than b=μb=\mu. While Appendix C.3 establishes that b=μb=\mu yields equiprobable responses under symmetry, the stronger optimality claim requires additional justification.

  2. Gaps in the proof of the optimal query direction.

    Appendix C.4 replaces expected conditional entropy minimization with maximization of the expected absolute response score without proving that the two objectives have the same optimizer. Furthermore, the argument for selecting the principal eigenvector appears to rely on the marginal distribution of u⊤Su^\top S, although the integrand also depends on ∥S∥2\|S\|^2. The joint dependence is not adequately addressed.

    Appendix C.5 also switches from optimizing the expected absolute response score to minimizing expected conditional entropy when determining the query magnitude, without establishing equivalence between the two objectives.

  3. Limited theoretical guarantees for discrete approximation.

    Pair M-dist relies on a second-order expansion around the synthesized query and assumes that the gradient of mutual information vanishes at this point. This assumption depends on the unestablished optimality result above. Neither approximation method provides an explicit guarantee on information loss after filtering.

    In addition, both methods still score candidate pairs, resulting in quadratic scaling with the number of available items in the general case.

  4. Limitations of the empirical validation.

    The human color study includes only 20 participants, and uncertainty estimates or statistical significance tests are not sufficiently reported. The calibration protocol and the use of past responses for predictive evaluation also require clarification.

    The synthetic experiments primarily evaluate the method under response models aligned with its assumptions. The model-mismatch experiments show that the advantage of Info-Synth over Gauss Search Synthesis diminishes under alternative response mechanisms.

    The Reddit experiments provide useful held-out evaluations, but cover only three users and do not directly validate the ideal-point assumption in the embedding space. The robot experiments use an automated oracle based on trajectory tracking error rather than actual human feedback.

  5. High-dimensional approximation and efficiency trade-offs.

    Appendix Figure 26 shows that the direction-alignment term of Pair Opt-dist can negatively affect performance in the 512-dimensional setting. In addition, the reported wall-clock comparisons would be strengthened by end-to-end time-to-target accuracy measurements, including posterior updates and query selection.

Questions for authors
  1. Can the authors provide a rigorous proof that b=μb=\mu globally maximizes the complete mutual-information objective, rather than only its marginal entropy term? How is the inference from a⊤(μ−b)=0a^\top(\mu-b)=0 to b=μb=\mu justified?

  2. Can the authors establish that maximizing the expected absolute response score is equivalent to minimizing expected conditional entropy? How does the proof of principal-eigenvector optimality account for the joint dependence on u⊤Su^\top S and ∥S∥2\|S\|^2?

  3. Have the authors numerically compared Info-Synth against direct optimization of the full mutual-information objective in low-dimensional settings? Such a comparison could help validate the claimed optimality.

  4. Is the negative Hessian used in Pair M-dist guaranteed to be positive semidefinite at the synthesized query? What happens if the synthesized point is not a stationary point of mutual information?

  5. How were response-model parameters calibrated in the human study, and were any calibration or evaluation responses shared? Could the authors report confidence intervals or participant-level variability?

  6. Could the authors report end-to-end computational costs at matched target accuracy, and clarify whether PCA fitting and hyperparameter selection in the Reddit experiments used only the permitted information?

  7. How sensitive is Pair Opt-dist to the weighting parameter η\eta, and could the authors further explain why the direction-alignment term hurts performance in the higher-dimensional experiments?

1 comment

The optimality concerns in W1–W2 and Q1–Q2 are addressed in our common response and the revised Section 4 and Appendix C.3 in the paper.

Further experiments on direct optimization of mutual information, and more details about Pair M-dist, user study, computational details and model mismatch are also addressed in the common response.

1. Reddit experiments (W4, W5, Q6)

Common Response gives full details. In brief, the Reddit experiments now cover 10 users instead of three (Figures 6 and 26–29), run in the full 512-dimensional embedding space without PCA, and report total query time over all 80 queries. Across all 10 users and both noise levels, Pair Opt-dist (γ=0.2\gamma=0.2) shows no loss in accuracy compared with Active Discrete: the curves are identical at every query in 17 of 20 settings, and the largest deviation is 1.4 percentage points. Its median total query time is 36.8% of Active Discrete's. Because the two methods reach each accuracy level after the same number of queries, this saving also holds for time-to-target accuracy (W5).

Use of permitted information (Q6). With PCA removed, nothing is fit to the embeddings. The held-out 30% of each user's responses is never used for query selection, posterior inference, or hyperparameter choice. We report both noise values, σ0=0.1\sigma_0=0.1 and σ0=0.3\sigma_0=0.3, rather than selecting one. The weights η2=η3=1\eta_2=\eta_3=1 and the filter level γ=0.2\gamma=0.2 were fixed in advance and not tuned on any data.

Ideal-point assumption (W4). We do not test the ideal-point assumption in the embedding space directly. The held-out prediction accuracy across 10 users is indirect evidence that it is a usable approximation for this task.

2. Pair Opt-dist in 512 dimensions and sensitivity to η\eta (W5, Q7)

We revised the Pair Opt-dist score (Eq. 9) to separate query magnitude from direction. In Figure 30, the full score now matches Active Discrete in 512 dimensions, so the direction-alignment term no longer degrades performance. On its own, however, the direction term is still weak in 512 dimensions.

Our single-term ablations explain why. The candidate query directions have very little alignment with the leading posterior eigenvector v1\mathbf{v}_1 across all 80 queries (Figure 31). As a result, the direction term barely varies across candidates (Figure 30d) and is only weakly correlated with MI (Figure 30c), so it cannot tell a good query from a poor one. We attribute this to how hard it is to estimate a dominant posterior direction in 512 dimensions from at most 80 responses. In contrast, the magnitude term, which favors queries at an informative separation, has a Spearman correlation with MI close to −1-1 and by itself matches Active Discrete in 512 dimensions.

Sensitivity to η\eta (Q7). We used η2=η3=1\eta_2=\eta_3=1 in all experiments and did not sweep these values. The single-term ablations correspond to the extreme settings. They suggest that each weight should reflect how well its term discriminates between candidates: a larger η2\eta_2 and a smaller η3\eta_3 in high dimensions, with relatively more weight on the direction term in low dimensions, where the posterior covariance is better estimated.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Revised Optimality Analysis

We thank the reviewers for the careful and constructive assessment and for recognizing the motivation and practical value of the framework. In this response, we address the concerns shared across the reviews. We will follow up over the next few days with responses to the additional comments and questions raised by individual reviewers.

We are particularly grateful for the detailed examination of the optimality argument in Sec. 4 and App. C. During the development of the paper, we changed the observation model to incorporate distance-dependent response uncertainty, but missed updating the theoretical justification consistently with that change. The reviewers correctly identified gaps in the original derivation.

Optimality of the synthesized query.

The criticisms are correct: (i) a⊤μ=τ\mathbf{a}^{\top}\boldsymbol{\mu}=\tau does not imply b=μ\mathbf{b}=\boldsymbol{\mu}; (ii) optimizing the two entropy terms separately does not by itself give a joint maximizer; and (iii) the proof of Prop. 1 replaced the entropy by the surrogate E∣f∣\mathbb{E}|f| and ignored the dependence between u⊤S\mathbf{u}^{\top}S and ∥S∥2\|S\|^2. We have rewritten this part of the paper.

What we now prove (new Theorem 1; complete proof in App. C.3, no approximation). For a Gaussian posterior and any symmetric link, the Info-Synth query μ±r∗v1\boldsymbol{\mu}\pm r^*\mathbf{v}_1 maximizes the full mutual information over all mean-cut queries, i.e., all pairs whose bisecting hyperplane contains the posterior mean (a⊤(μ−b)=0\mathbf{a}^{\top}(\boldsymbol{\mu}-\mathbf{b})=0, which is the condition appearing in our original derivation). This class contains every pair centered at μ\boldsymbol{\mu}, in any direction and with any separation, as well as pairs displaced within the hyperplane. The new proof addresses each concern directly:

  • Full objective. We use I≤1−H(Y∣W)I\leq 1-H(Y\mid W), with equality for centered queries, so the marginal term is never optimized in isolation.
  • No surrogate. The expected conditional entropy is compared directly, by a stochastic-ordering argument that is valid for any function that decreases in ∣f∣|f|.
  • Joint dependence on u⊤S\mathbf{u}^{\top}S and ∥S∥2\|S\|^2. We condition on u⊤S\mathbf{u}^{\top}S and compare the conditional law of the perpendicular component (Anderson’s inequality and eigenvalue interlacing).
  • Direction and separation are optimized jointly; what remains is exactly the one-dimensional search of Alg. 1 (the factor-of-4 inconsistency between a=rv1\mathbf{a}=r\mathbf{v}_1 and a=4rv1\mathbf{a}=4r\mathbf{v}_1 was a typo and is fixed).

We also prove that the synthesized query is a stationary point of the mutual information over all pairs (p,q)(\mathbf{p},\mathbf{q}), which is the property the Taylor expansion behind Pair M-dist relies on, and that the Hessian there is negative semidefinite on a (2d−1)(2d-1)-dimensional subspace; App. C.4 provides an expression for the curvature along the remaining direction, with numerical checks for d≥2d\ge2 planned for the next revision.

What we no longer claim. We do not claim optimality over all queries as a theorem. The mean-cut restriction mirrors the mean-cut/equiprobable design principle of Canal et al. (2019).

A statement over all queries (new Theorem 2; proof in App. C.5). Without the mean-cut restriction we can still show the following. Under the Gaussian posterior every candidate user point has an equally likely mirror image on the other side of the posterior mean, and part of what a response reveals is on which side of the mean the user lies. We define the separation information of a query as the information about this that is due to mirror-image points preferring different items, evaluated at the reliability of the less reliable of the two. It is a lower bound on the mutual information of every query, it equals the mutual information of every query centered at the posterior mean, and the Info-Synth query maximizes it over all pairs (p,q)∈Rd×Rd(\mathbf{p},\mathbf{q})\in\mathbb{R}^d\times\mathbb{R}^d, in every dimension and for every symmetric link. Thus, Theorem 2 establishes global optimality for separation information over all query pairs. We will complement this guarantee with numerical comparisons against unrestricted mutual information optimization and report the results in the next revision.

Thanks for the rebuttal. To me, it seems the paper's main result has now changed significantly, diluting its contribution. Hence, I maintain my original rating.

AuthorsOpenReview ↗

Addressing common concerns

1. Direct optimization of the full mutual information objective

To assess Info-Synth without the mean-cut restriction, we directly optimize the full mutual-information estimate I(W;Y∣p,q)I(W;Y\mid \mathbf{p},\mathbf{q}) over the 2d2d query coordinates using L-BFGS. The results are described in Appendix F.2.3. We compared the performance of Info-Synth and direct MI optimization for σ0=[0.001,0.01,0.05,0.1,0.2,0.5]\sigma_0 = [0.001, 0.01, 0.05, 0.1, 0.2, 0.5].

2. Pair M-dist: score interpretation and approximation

Pair M-dist uses a quadratic score based on M=−H(z⋆)\mathbf{M}=-\mathbf{H}(\mathbf{z}^\star). In practice, I(z∗)I(\mathbf{z}^*) is indeed greater than I(z)I(\mathbf{z}) for any z\mathbf{z} in the dataset. For a non-Gaussian posterior or finite-sample MI objective, MM can be indefinite and produce negative scores, which remain sortable. We have included this discussion in Section 5 and Appendix F.2.3.

Despite this, the filter retains informative pairs, and the refinement stage evaluates their MI to select the best pair. We computed the the selected pair’s relative MI loss compared with the highest-MI candidate pair for D=10D=10 and N=[100,500]N=[100, 500]. The median information loss over 55 trials for N=100N=100 was under 6%6\% and that for N=500N=500 was under 4%4\%.

3. Computational details

Filtering scores the PP allowable pairs, so P=Θ(N2)P=\Theta(N^2) when all pairs are permitted. With SS posterior samples in dd dimensions and ρ\rho retained fraction, an illustrative estimate of the cost relative to exhaustive MI evaluation is ρ+d/S+d/P\rho+d/S+d/P, assuming equal leading constants. With S=4000S=4000 and P=4950P=4950 (N=100N=100) candidate pairs, retaining 20% or 60% gives approximately 80% or 40% less computational work for d≤10d\leq10; at d=512d=512, the corresponding reductions are approximately 57% and 17%. The approximation saves work when ρ+d/S+d/P<1\rho+d/S+d/P<1, with larger candidate pools amortizing setup more effectively. These estimates describe query-selection work; actual runtime gains depend on implementation constants and additional overheads.

4. Human-study objective, calibration, and evaluation

Our goal is accurate preference estimation through adaptive comparisons, so each response model is evaluated jointly with query selection and inference. Only the initial comparison was common among all response models; subsequent comparisons used the same MI criterion under each model's current posterior. Cross-entropy and choice accuracy retrospectively describe each condition's collected responses under that posterior, complementing preference estimation error.

The noise parameter σ0\sigma_0 was calibrated using one user's responses before the study and reused for the participants in the study without refitting. The same values were able to explain the study participants' responses well on an average. We have explained this protocol clearly in Appendix F.1.

The mean confidence ratings summarize participants' self-reported confidence during the study. We have updated the confidence rating figure to report participant level variability. For each method, we first averaged the confidence ratings across queries within each participant. The bars show the mean of these 20 participant averages, and the whiskers show their 25th and 75th percentiles. This gives each participant equal weight and shows how average confidence varied across participants.

5. User model mismatch

The mismatch experiments test alternative response mechanisms. At a fixed hyperplane, separation does not affect normalized model response probabilities, whereas larger gaps can improve plain model reliability. These mechanisms lack the finite-separation trade-off targeted by Info-Synth, consistent with its similar performance to Gauss Search Synthesis in Figures 19–20. Active Discrete achieves lower errors in these settings at greater computational cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.