- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies recovery of a functional of an unknown latent population from responses to selected probes. It connects the largest target disagreement among observationally equivalent populations to uniform approximation error and minimax recovery error. This motivates Functional Ambiguity Reduction, which selects queries according to the remaining target ambiguity. A finite-batch procedure combines a worst-branch constraint with worst-population expected midpoint error. Experiments consider population-response recovery, controlled observational equivalence, and limited-budget LLM evaluation.
Critical strengths
The connection between observational equivalence, approximation error, and minimax recovery is precise. Distinguishing ambiguity within a realized compatible set from worst-case ambiguity over a design gives a useful interpretation of recoverability.
The finite-batch objective keeps response probabilities and target values coupled through the same population. The ablations examine this component, and the Duolingo-FR and LLM results provide positive empirical evidence.
Critical weaknesses
The empirical target is not sufficiently connected to the theoretical target. Appendix C.1 defines using fitted response-kernel predictions, while empirical errors are evaluated against held-out responses. Appendix B.2 addresses acquisition-response discrepancy and sieve approximation, but these do not automatically account for estimation error in . The fitted model and discrepancy-radius calibration are not described sufficiently to verify this connection.
The gains over the closest comparator can be small, while their computational cost is unclear. Against safety-matched TSIED, Appendix D.2 reports expected errors of 28.5190 versus 28.5442 on OpinionQA and 31.4794 versus 31.5425 on CES. FAR also does not achieve the lowest MAE on ASSISTments. Runtime and uncertainty for these paired improvements would help establish the practical benefit.
Questions for authors
-
In the real-response experiments, Appendix C.1 constructs from fitted response-kernel predictions. How is the difference between this fitted and the target function for actual population performance accounted for in the reported intervals? Appendix B.2 addresses acquisition-response discrepancy and sieve approximation, but this additional error is not clear.
-
For the Binomial experiments, what response-model fitting procedure and data-based rule for the discrepancy radius were used? Please also report empirical coverage and the frequency and magnitude of the additional feasibility relaxation in Eq. (23).
-
What per-query runtimes and achieved selection-gap certificates were observed for the Binomial solver? For the Hypergeometric allocation search in Appendix C.5, how much do the selected allocations and estimation errors vary across independent search initializations?
Thank you for your comments and questions.
W1 & Q1. Both targets concern population-average performance on the same prespecified items Rh. The actual target uses true response probabilities; Appendix C.1 approximates it by h from fitted kernels. Thus, h approximates the same functional, not a separate evaluation objective. CIFAR-10H/Duolingo-FR evaluate recovery on actual responses from independent held-out respondents, not fitted predictions. Fitting, calibration, acquisition and evaluation use disjoint respondent roles; model parameters are fixed before acquisition. This directly tests whether model-guided acquisition improves recovery of actual population performance. Moreover, FAR's treatment of response-model uncertainty naturally extends to the target function itself. If model-error bounds hold for reference items, with ∥kr⋆−kr∥∞≤ρr, linearity yields ∣Th⋆(F)−Th(F)∣≤∣Rh∣−1∑r∈Rhρr. When the true population remains compatible, the fitted-target interval [Ls,Us] covers actual performance after expanding both endpoints by this bound. Together with Appendix B.2's acquisition-response uncertainty, this links model error, population compatibility and actual target recovery without changing FAR's framework. We will clarify this propagation.
W2. Table 1 gives FAR the best mean on all six Duolingo-FR metrics: MAE 5.0% below the strongest baseline and 7.0% below TSIED, plus P95 9.2% below TSIED. CIFAR-10H has the lowest MAE, significant after correction (p=0.0128). In Fig. 4, five FAR queries (MAE 0.12869) beat six baseline queries (0.13303), saving 16.7% of responses at equal batch size. Fig. 3 shows 7.0%/25.7% lower full-score MAE on Qwen3-4B/Phi-3.5-mini with 19 items. Eq. (11) does not guarantee these gains. For the reviewer's safety-matched TSIED comparison, we further examine paired uncertainty and query-selection differences. On CES, query selections differ in 115/200 (57.5%) states at B=10. Fig. 2 shows FAR cuts target-conflicting population discrimination error from 1.07% to 0.50% (53.3%) at 40 responses. This shows different decisions and better target discrimination alongside held-out recovery gains. Sec. 5.1 discusses ASSISTments' width–MAE trade-off; Q2 addresses model limitations. Q3 reports runtime and solver certificates.
Q2. CIFAR/Duolingo separate fitting, calibration, acquisition and evaluation roles. Standardized accuracies define 121-point latent support on [−3,3]; clipped item-accuracy logits yield kθ(1∣x,z)=σ(ax+z). Each split preselects the corrected 95th percentile of absolute calibration residuals (rank ⌈0.95(ncal+1)⌉) as its radius. Radii are 0.1986–0.2161/0.2473–0.3479 and enter compatibility and risk (Eqs. 15–16).
The first two achieve full reference coverage without repair/fallback and show recovery gains. ASSISTments shows the Sec. 5.1 compatibility issue. Positive Eq. (23) relaxation averages 0.26145 (max 0.56378); all 114 abnormal width increases arise from incompatibility-triggered shared full-range fallback. Still, FAR has the lowest normalized width and highest CCR.
Q3. The Binomial solver (Eqs. 20–22) uses 2D response–target optimization. Times (s/query, excluding fitting) and max selection gaps:
FAR requires more compute per query but can recover better with fewer queries. Fig. 4 shows five FAR queries (MAE 0.12869) beat six strongest-baseline queries (0.13303) on Duolingo-FR, saving 16.7% of online responses. This trades selection compute for response efficiency; end-to-end time gains depend on response-collection costs. For Hypergeometric allocation, three seeds reoptimized 11 count states across 120 model–history records (48 proposals and one local search per state). Full-score MAE (pp) and changed allocations out of 60 histories:
Despite allocation changes, the worst seeds still have 4.9%/19.9% lower MAE than the strongest baselines. The Hypergeometric search shows empirical stability across seeds; the Binomial solver provides numerical selection certificates.