acceptodds
Under review as a conference paper at ICLR 2027

Functional Ambiguity Reduction: Minimax Acquisition under Observational Equivalence

Abstract

Finite observations can leave distinct latent populations observationally indistinguishable while those populations imply different values of a target functional. We study how this target-relevant ambiguity limits recovery and how it should guide subsequent acquisition. We first characterize recoverability under a finite response design: the largest functional ambiguity is exactly twice the best uniform approximation error of the target by the observed response coordinates, and this approximation error equals the minimax recovery error of any estimator based only on the observed response vector. This characterization yields Functional Ambiguity Reduction (FAR), which selects queries according to the target disagreement that can remain among observationally compatible populations. FAR is one-step minimax-optimal for worst-case absolute target recovery under exact responses and extends to finite response batches and learned response models. Across five public datasets, FAR shows its clearest gains on CIFAR-10H and Duolingo-FR, reducing MAE on Duolingo-FR from \(0.1330\) for the strongest baseline to \(0.1263\). We further apply FAR to budgeted LLM evaluation, where five additional queries reduce score-estimation MAE by on Qwen3-4B and on Phi-3.5-mini relative to the strongest baseline for each model. These results connect finite-design identifiability directly to adaptive acquisition for population-level functional recovery. Code is available at https://anonymous.4open.science/r/FAR-D5B5/.

Primary area:
learning theory
Keywords:
Observational equivalence, Partial identification, Adaptive acquisition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_4ZYaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recovery of a functional of an unknown latent population from responses to selected probes. It connects the largest target disagreement among observationally equivalent populations to uniform approximation error and minimax recovery error. This motivates Functional Ambiguity Reduction, which selects queries according to the remaining target ambiguity. A finite-batch procedure combines a worst-branch constraint with worst-population expected midpoint error. Experiments consider population-response recovery, controlled observational equivalence, and limited-budget LLM evaluation.

Critical strengths

The connection between observational equivalence, approximation error, and minimax recovery is precise. Distinguishing ambiguity within a realized compatible set from worst-case ambiguity over a design gives a useful interpretation of recoverability.

The finite-batch objective keeps response probabilities and target values coupled through the same population. The ablations examine this component, and the Duolingo-FR and LLM results provide positive empirical evidence.

Critical weaknesses

The empirical target is not sufficiently connected to the theoretical target. Appendix C.1 defines hh using fitted response-kernel predictions, while empirical errors are evaluated against held-out responses. Appendix B.2 addresses acquisition-response discrepancy and sieve approximation, but these do not automatically account for estimation error in hh. The fitted model and discrepancy-radius calibration are not described sufficiently to verify this connection.

The gains over the closest comparator can be small, while their computational cost is unclear. Against safety-matched TSIED, Appendix D.2 reports expected errors of 28.5190 versus 28.5442 on OpinionQA and 31.4794 versus 31.5425 on CES. FAR also does not achieve the lowest MAE on ASSISTments. Runtime and uncertainty for these paired improvements would help establish the practical benefit.

Questions for authors
  1. In the real-response experiments, Appendix C.1 constructs hh from fitted response-kernel predictions. How is the difference between this fitted hh and the target function for actual population performance accounted for in the reported intervals? Appendix B.2 addresses acquisition-response discrepancy and sieve approximation, but this additional error is not clear.

  2. For the Binomial experiments, what response-model fitting procedure and data-based rule for the discrepancy radius ρx\rho_x were used? Please also report empirical coverage and the frequency and magnitude of the additional feasibility relaxation in Eq. (23).

  3. What per-query runtimes and achieved selection-gap certificates were observed for the Binomial solver? For the Hypergeometric allocation search in Appendix C.5, how much do the selected allocations and estimation errors vary across independent search initializations?

1 comment

Thank you for your comments and questions.

W1 & Q1. Both targets concern population-average performance on the same prespecified items RhR_h. The actual target uses true response probabilities; Appendix C.1 approximates it by hh from fitted kernels. Thus, hh approximates the same functional, not a separate evaluation objective. CIFAR-10H/Duolingo-FR evaluate recovery on actual responses from independent held-out respondents, not fitted predictions. Fitting, calibration, acquisition and evaluation use disjoint respondent roles; model parameters are fixed before acquisition. This directly tests whether model-guided acquisition improves recovery of actual population performance. Moreover, FAR's treatment of response-model uncertainty naturally extends to the target function itself. If model-error bounds hold for reference items, with ∥kr⋆−kr∥∞≤ρr\lVert k_r^\star-k_r\rVert_\infty\le\rho_r, linearity yields ∣Th⋆(F)−Th(F)∣≤∣Rh∣−1∑r∈Rhρr|T_{h^\star}(F)-T_h(F)|\le |R_h|^{-1}\sum_{r\in R_h}\rho_r. When the true population remains compatible, the fitted-target interval [Ls,Us][L_s,U_s] covers actual performance after expanding both endpoints by this bound. Together with Appendix B.2's acquisition-response uncertainty, this links model error, population compatibility and actual target recovery without changing FAR's framework. We will clarify this propagation.

W2. Table 1 gives FAR the best mean on all six Duolingo-FR metrics: MAE 5.0% below the strongest baseline and 7.0% below TSIED, plus P95 9.2% below TSIED. CIFAR-10H has the lowest MAE, significant after correction (p=0.0128p=0.0128). In Fig. 4, five FAR queries (MAE 0.12869) beat six baseline queries (0.13303), saving 16.7% of responses at equal batch size. Fig. 3 shows 7.0%/25.7% lower full-score MAE on Qwen3-4B/Phi-3.5-mini with 19 items. Eq. (11) does not guarantee these gains. For the reviewer's safety-matched TSIED comparison, we further examine paired uncertainty and query-selection differences. On CES, query selections differ in 115/200 (57.5%) states at B=10B=10. Fig. 2 shows FAR cuts target-conflicting population discrimination error from 1.07% to 0.50% (53.3%) at 40 responses. This shows different decisions and better target discrimination alongside held-out recovery gains. Sec. 5.1 discusses ASSISTments' width–MAE trade-off; Q2 addresses model limitations. Q3 reports runtime and solver certificates.

Q2. CIFAR/Duolingo separate fitting, calibration, acquisition and evaluation roles. Standardized accuracies define 121-point latent support on [−3,3][-3,3]; clipped item-accuracy logits yield kθ(1∣x,z)=σ(ax+z)k_\theta(1|x,z)=\sigma(a_x+z). Each split preselects the corrected 95th percentile of absolute calibration residuals (rank ⌈0.95(ncal+1)⌉\lceil0.95(n_{cal}+1)\rceil) as its radius. Radii are 0.1986–0.2161/0.2473–0.3479 and enter compatibility and risk (Eqs. 15–16).

DatasetReference coverageλ>0\lambda>0Fallback
CIFAR-10H90/900/3600/90
Duolingo-FR90/900/3600/90
ASSISTments61/210186/840114/210

The first two achieve full reference coverage without repair/fallback and show recovery gains. ASSISTments shows the Sec. 5.1 compatibility issue. Positive Eq. (23) relaxation averages 0.26145 (max 0.56378); all 114 abnormal width increases arise from incompatibility-triggered shared full-range fallback. Still, FAR has the lowest normalized width and highest CCR.

Q3. The Binomial solver (Eqs. 20–22) uses 2D response–target optimization. Times (s/query, excluding fitting) and max selection gaps:

DatasetFAR / TSIED (s)Max selection gap
CIFAR-10H5.50 / 1.924.32×10−64.32\times10^{-6}
Duolingo-FR5.11 / 1.398.67×10−68.67\times10^{-6}
ASSISTments8.80 / 3.431.84×10−61.84\times10^{-6}

FAR requires more compute per query but can recover better with fewer queries. Fig. 4 shows five FAR queries (MAE 0.12869) beat six strongest-baseline queries (0.13303) on Duolingo-FR, saving 16.7% of online responses. This trades selection compute for response efficiency; end-to-end time gains depend on response-collection costs. For Hypergeometric allocation, three seeds reoptimized 11 count states across 120 model–history records (48 proposals and one local search per state). Full-score MAE (pp) and changed allocations out of 60 histories:

InitializationQwen MAE (changes)Phi MAE (changes)
Original6.9015 (—)5.3437 (—)
3900007.0533 (28/60)5.1885 (14/60)
4900006.8788 (13/60)5.3414 (7/60)
5900006.7552 (27/60)5.7590 (27/60)
Strongest baseline7.41737.1891

Despite allocation changes, the worst seeds still have 4.9%/19.9% lower MAE than the strongest baselines. The Hypergeometric search shows empirical stability across seeds; the Binomial solver provides numerical selection certificates.

Reviewer_vHdGOpenReview ↗
Rating
2(+1)
Confidence
2(−1)
Informativeness
2(+1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper asks how to choose the next query when the data collected so far cannot tell apart several latent populations that disagree on a target quantity. It characterizes how large this ambiguity can be and proposes FAR, which picks the query that leaves the least worst-case ambiguity. Experiments cover crowdsourcing, language learning, education, survey, and LLM evaluation data.

Critical strengths

The question is well motivated: the paper focuses on the ambiguity that matters for the target, rather than on all remaining uncertainty.

Critical weaknesses

W1. The theory is mostly known. Theorem 1 says that the midpoint of an interval is the best guess, which is trivial. Theorem 2 is a classical result from optimal recovery (Micchelli and Rivlin, 1977; Traub, Wasilkowski and Woźniakowski, 1988) applied to probability distributions, and the paper does not position itself against this literature. The method is essentially a worst-case version of TSIED, and its optimality holds by definition and for one step only. I could not find a new idea beyond combining these pieces.

W2. The experiments do not show that FAR works better in practice. Much of the evaluation, especially Table 2, scores every method by FAR's own objective inside the fitted model, so FAR wins by design, and even then the margins are tiny. The approach also relies on a response model that is fitted in advance and treated as exact. On ASSISTments, where this model does not seem to fit the data, the intervals widen instead of shrinking, FAR's error grows with more queries, and a baseline does better.

Questions for authors
  1. Does adaptive selection actually help here, compared with a fixed design chosen in advance?
  2. What happens when the response model is wrong?
1 comment

Thank you for your comments and questions.

W1. FAR establishes a recovery-to-acquisition framework for population functionals under finite observations. Theorem 1 gives fibrewise minimax error as half the compatible target width. Theorem 2 gives design-wide ambiguity as ΔM(h)=2dist⁡∞(h,VM)\Delta_M(h)=2\operatorname{dist}_\infty(h,V_M). We clarify the classical connection: Micchelli–Rivlin (1977, Eq. 21) gives annihilator duality, while Traub et al. (1988, Thm. 5.5.1) studies balanced convex classes. Our probability-distribution class is non-balanced. Appendix A.2 uses Jordan decomposition to construct valid observationally equivalent populations attaining the bound. Appendix A.3 proves uniform minimax recovery, an attaining response-affine estimator, and identifiability iff h∈VMh\in V_M. More importantly, FAR turns these limits into adaptive acquisition. With exact responses, FAR minimizes residual ambiguity, twice the worst post-query minimax error. For finite batches, Eqs. (9)–(11) combine worst-branch minimax safety with worst-population expected midpoint error. The same compatible population determines response probabilities and target values, preserving their coupling beyond width minimization. This produces different decisions from TSIED. In Fig. 7, safety-matched TSIED/FAR have expected widths of 70.41/72.37 percentage points (pp), yet recovery risks of 37.87/32.81 pp, a 5.06 pp FAR gain. Across 200 CES states, queries differ in 115/55/55 cases (B=10/40/160B=10/40/160). A worst-population expected-width ablation leaves 63/10/12 certified strict tradeoffs. Eqs. (20)–(22) reduce risk to certified 2D response–target optimization, with selection gaps below 9×10−69\times10^{-6}. The guarantee is one-step; FAR replans after each batch. Q1 shows 4.5%/11.5% lower MAE than target-aware fixed designs.

W2. FAR's practical effectiveness is evaluated beyond its optimization objective. Table 2 includes width and endpoint metrics beyond Eq. (11). Table 1 evaluates held-out response errors, while Fig. 3 measures score-recovery errors against the full 560-item LLM benchmark. Neither is FAR's fitted-model worst-case risk. Duolingo-FR MAE falls from 0.1330 to 0.1263 (5.0%) vs. the strongest baseline. On CIFAR-10H, FAR has the lowest MAE, significantly improving over TSIED after multiplicity correction. Fig. 4 shows that five FAR queries outperform six baseline queries, saving 16.7% of the budget. With 19 items, FAR improves Qwen3-4B/Phi-3.5-mini score-recovery MAE by 7.0%/25.7% against their strongest baselines. As discussed in Sec. 5.1, ASSISTments shows the trade-off between functional ambiguity reduction and empirical point error. The 114 marked width increases arise from incompatibility-triggered shared full-range fallback, while nonempty, unrelaxed fibres remain nested. Importantly, FAR still achieves the lowest normalized width and highest CCR, demonstrating effective target-ambiguity reduction despite model-compatibility challenges. Appendix B.2.1 and Q2 further address model discrepancy and calibration.

Q1. To evaluate adaptive acquisition, we added two fixed-design baselines: random selection and a target-aware fixed design that greedily minimizes the uniform approximation error in Theorem 2. Unlike FAR, which updates its compatible set and selects queries using observed responses, the fixed designs preselect all four queries.

DatasetRandom MAETarget-aware MAEFAR MAEGain
CIFAR-10H0.054180.041350.039504.5%
Duolingo-FR0.145810.142700.1263211.5%

FAR reduces final MAE by 4.5%/11.5% versus target-aware fixed design on CIFAR-10H/Duolingo-FR. These results show FAR's response-adaptive benefit beyond target-aware query preselection.

Q2. FAR handles model discrepancy: Eqs. (15)–(16) incorporate calibrated radii ρx\rho_x into compatible sets and risks. With valid joint model/sampling bounds and adequate latent representation, Eq. (12) ensures across-round containment of at least 1−α−β1-\alpha-\beta. CIFAR-10H/Duolingo-FR use independently calibrated nonzero radii, each giving 90/90 reference coverage without fallback. We tested 0.5×/1×0.5\times/1\times radii over four rounds, five fixed splits and three targets, holding fitted models and responses fixed.

DatasetRadiusFAR MAETSIED MAEFAR width
CIFAR0.50.026120.026260.13459
CIFAR1.00.036440.036870.15819
Duolingo0.50.071440.076050.41471
Duolingo1.00.113820.130700.51990

Wider allowances increase interval conservatism, revealing a robustness–precision tradeoff.

Comments on the paper

1 comment

Thank you for your comments and questions.

W1&Q1. Closest are Micchelli–Rivlin (1977) on annihilator duality and Traub et al. (1988) on balanced convex classes. Our unknown is a probability distribution F∈P(Z)F\in\mathcal P(\mathcal Z), which is convex but not balanced, and observations are characterized by expectations of response-kernel coordinates under FF. Appendix A.2 proves both directions: differences of equivalent populations annihilate VM=span⁡{1,κM,j}V_M=\operatorname{span}\lbrace1,\kappa_{M,j}\rbrace; conversely, annihilating signed measures of total variation at most two yield equivalent populations via Jordan decomposition. Thus ΔM(h)=2dist⁡∞(h,VM)\Delta_M(h)=2\operatorname{dist}_\infty(h,V_M) is attainable, not just an abstract dual bound. Appendix A.3 gives minimax error dist⁡∞(h,VM)\operatorname{dist}_\infty(h,V_M), an attaining response-affine estimator and identifiability iff h∈VMh\in V_M. Theorem 1 gives fibrewise error WM(pM)/2W_M(p_M)/2. Fixed-design duality does not yield FAR's finite-batch acquisition. Eqs. (9)–(11) combine worst-branch recovery safety with worst-population expected midpoint loss, preserving response–target coupling within one population. Eqs. (20)–(22) reduce risk to certified 2D polynomial optimization, with gaps below 9×10−69\times10^{-6} in all three Binomial tasks. We will clarify this.

W2&Q2. FAR's benefits extend to average-case target recovery. Table 2's RTE, efficiency regret, and oracle agreement validate acquisition, while Fig. 5 evaluates MAE against prespecified population truth. FAR has the lowest MAE of ten methods on OpinionQA and CES; MSE analysis shows 5.95%/2.53% reductions. We added held-out-response evaluation across all six two-item average targets on OpinionQA, with exact averaging over future response sampling. FAR reduces actual recovery MAE by 7.98% and MSE by 11.38% vs. TSIED. This shows joint response–target optimization yields average-case gains beyond the worst-case objective. Real-response experiments further support these gains. Table 1 shows FAR leads all six Duolingo-FR metrics, with 5.0% lower MAE than the strongest baseline, 7.0% lower MAE and 9.2% lower P95 than TSIED. CIFAR-10H has the lowest MAE, significant after multiplicity correction (p=0.0128p=0.0128). Fig. 3 reports 7.0%/25.7% lower MAE on actual 560-item LLM scores, while Fig. 4 shows 16.7% fewer responses to surpass the strongest baseline on Duolingo-FR.

Q3. FAR accounts for response-model uncertainty (Eqs. 15–16). CIFAR-10H/Duolingo-FR use disjoint fitting, calibration, acquisition, and evaluation roles, with 121-point logistic latent kernels. Each split fixes its radius before acquisition at the corrected 95th percentile of calibration residuals, yielding 0.1986–0.2161/0.2473–0.3479. Both attain 90/90 held-out reference coverage without relaxation (0/360) or fallback (0/90). Sec. 5.1 discusses ASSISTments model compatibility. Diagnostics show 61/210 coverage, 186/840 positive relaxations (mean/max 0.26145/0.56378) and 114/210 fallbacks. All abnormal widenings arise from shared infeasibility fallback. FAR has the lowest normalized width and highest CCR. We evaluate radius sensitivity over four rounds, five fixed splits and three targets per dataset:

DatasetRadiusFAR MAETSIED MAEFAR width
CIFAR-10H0.5×0.026120.026260.13459
1.0×0.036440.036870.15819
Duolingo-FR0.5×0.071440.076050.41471
1.0×0.113820.130700.51990

All retain 15/15 reference coverage without fallback, showing a robustness–precision trade-off. Reducing the latent sieve from 121 to 61 points preserves 56/60 (93.3%) selections and 12/15 complete sequences per dataset, with similar MAE/width.

Q4. FAR differs from TSIED width minimization when interval narrowing does not imply lower target-recovery error. TSIED prioritizes reducing target bounds, while FAR minimizes expected midpoint error, coupling response probabilities and target values through the same compatible population. This matters with similar safety but different response–target relationships. In Fig. 7, under identical safety, FAR's expected width is 72.37 vs. 70.41 pp for TSIED, yet worst-population expected error falls from 37.87 to 32.81 pp (13.4%). On CES (B=10B=10), queries differ in 115/200 states. To isolate FAR's acquisition objective, we introduce safety-matched TSIED with the same worst-branch safety constraints and other four-round settings:

DatasetOriginal TSIED MAESafety-matched TSIED MAEFAR MAE
CIFAR-10H0.039940.039730.03950
Duolingo-FR0.135800.135800.12632

Under identical safety, FAR achieves 7.0% lower held-out MAE, 9.2% lower P95, and 3.3% lower Err-AUC on Duolingo-FR. FAR also reaches 0.12869 MAE with 50 responses, beating safety-matched TSIED's 0.13580 with 60, saving 16.7% of responses. Thus, the different objectives yield measurable recovery and budget-efficiency gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.