acceptodds
Under review as a conference paper at ICLR 2027

When Retrieval Endpoint Scores Miss the Value of Feedback

Abstract

Equal blind and known-condition retrieval scores do not certify equal partial-feedback value. An exact average-precision construction matches dimension, action count, and endpoints while reversing the preferred question. A natural-image submenu census finds disjoint optimal one-question sets in 13 of 1,920 cells; six also favor a paid question over stopping, and one native-rank rational-AP case verifies the reversal. This evaluator-only existence result does not establish a deployable selector. Across the same learned libraries, endpoint-preserving deletion preserves every optimal tree in 1,887 cells, including 318 of 351 with positive allowed-probe risk. Publicly predicted endpoint cores show no established replanning advantage over matched menus. At exactly two answers, SOURCE-calibrated planning improves single-ranker AP over SOURCE-selected controls, while fusion gains remain unestablished. Endpoint insufficiency, retained-action inadequacy, and deployed-policy failure are distinct diagnoses. These exposed-population results support evaluation under explicit information and cost assumptions, not a universal repair or serving speedup.

Primary area:
learning theory
Keywords:
conditional image retrieval, relevance feedback, value of information, adaptive querying, action pruning, average precision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tXZNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper attempts to demonstrate that standard benchmarks can be misleading. Showing that a system performs equally well on standard retrieval tests (whether tested "blind" or with known conditions) does not mean it gives equal value when asking users for clarification or feedback. The paper shows a mathematical example where two systems look identical on paper, but when you ask a clarifying question, the system that seemed worse actually becomes the better choice.

Critical strengths

None

Critical weaknesses

-- The paper is extremely difficult to read and follow and uses heavy jargon. It is not accessible to a general ML audience. I had to look up the meanings of various terms in almost every paragraph. Even the abstract was quite inaccessible.

-- The contribution is more of a statistical nature rather than core ML. There doesn't seem to be a direct machine learning use case or learning scenario involved. It is centred around statistical evaluation of a retrieval system, which could apply to any statistical evaluation of a system not specific to machine learning applications. The paper's relevance to ICLR is unclear.

Questions for authors

Could you provide use-cases directly relevant to AI/ML?

Could you explain the contribution and its significance over related works?

4 comments
AuthorsOpenReview ↗

Response to Reviewer tXZN

What machine-learning decision does this affect?

Consider an interactive image-retrieval system with several trained rankers. Its designer must decide which rankers to retain before deploying the system with feedback. A ranker can be unnecessary before any answer and when the desired condition is fully known, yet become useful after only one answer. Removing it solely because the two endpoint scores are preserved can therefore remove an intermediate decision that feedback makes useful.

Our exact average-precision construction shows that endpoint scores do not determine partial-feedback value. Two systems have the same dimension, five actions, blind AP of 239/360, and fully informed AP of one, yet prefer opposite questions. The useful ranking appears after partial information, so it need not be best at either endpoint. This is a realizable ranking example, not only an abstract payoff table. It does not imply a paid advantage at every budget and cost; at the studied two-question cap and cost 0.005, that example's paid advantage is zero.

When does deleting a learned ranker matter?

The next question is whether removing an action damages a useful question path. In the natural-image study, deletion preserves every optimal question tree in 1,887 of 1,920 cells. In 318 of the 351 cells with positive allowed-probe risk, it still preserves every optimal question tree. A possible loss therefore does not by itself show that the ranker is needed by an optimal feedback policy. There are thirteen endpoint-matched question reversals and six paid cases. Endpoint preservation alone cannot justify deletion, and possible-path loss alone cannot justify restoring a ranker: the relevant check is whether useful question paths retain their value. Sections 3–5 and Appendix A give this criterion; the separate restricted-width theorem and its assumptions remain in Appendix C.

In a separate CUB check with the fitted actor frozen, we examine query–library cases where the best achievable feedback value is unchanged. Terminal-menu restriction causes a loss above 0.2 AP points in 16/765 such cases at 32 dimensions and 16/754 at 128 dimensions, with questions, answers, STOP decisions and purchase counts unchanged (256 queries × three libraries per width). Restriction improves mean utility in both widths, so this identifies failures of individual fitted-policy preservation rather than average harm. The core is evaluator-derived; the result motivates reevaluating the fitted actor after a menu change, not a claim of a deployable pruner (Appendix J).

What completed learning experiment adds to the diagnosis?

A separate experiment tests a concrete learned-component choice. With the CLIP retrieval library and candidate policy menu fixed, replacing the DINO feedback observer with the CLIP observer improves two-answer AP by 0.67 points [0.16, 1.18]. Policies are selected on separate calibration queries and use only the query image and answers already obtained. The adjusted 95% interval resamples observer-training runs and query components separately while accounting for multiple comparisons. The clearer component gain is in using feedback to rank images; changing question selection alone remains inconclusive.

This supports observer selection under feedback. It does not turn the rare endpoint-matched reversals into a common failure mode or establish a new acquisition-learning method. The comparisons retain the documented benchmark splits and synthetic response assumptions; real human costs were not measured.

Interactive-retrieval methods provide mechanisms for acquiring and using feedback. We study what evaluation establishes about the learned system: an AP-realizable endpoint counterexample, a criterion connecting ranker deletion to preservation of optimal question paths, and a controlled observer-selection experiment. The results distinguish endpoint accuracy, preservation of feedback decisions, and performance of the fitted system. Value-of-information planning and rank fusion are existing tools; Section 2 explains this distinction from relative-attribute and dialog-feedback methods. Sections 3–8, Table 6, and Appendices A, J, and O connect the results to these ML decisions.

Dear Reviewer tXZN,

Our response gives two ML decisions: which trained image rankers to retain when users provide feedback, and which learned model should interpret that feedback. Do these examples clarify the ML use case?

Separately, our evaluation question is whether equal scores before feedback and with complete information ensure equal value after partial feedback. Does this clarify the contribution beyond methods for acquiring and using feedback? If not, which part of that distinction needs a clearer explanation?

Thank you for your time.

AuthorsOpenReview ↗

Which trained rankers should an image search system use for each query?

Dear Reviewer tXZN,

You asked for a use case that a machine-learning practitioner would recognize. Suppose an image search system has eight trained rankers. For a new query image, should it use all eight, or would a group of four work better once the user can answer a question? We tested a rule that reads all eight rankers’ predictions and chooses the full set or one of two groups of four before the first answer. The system can then ask a question or stop.

Compared with always using the full set, choosing the group per image increased average precision by 0.140 and 0.175 points on the 384- and 128-query SUN panels and used 0.113 and 0.059 fewer expected answers per query, respectively. These averages include every query. Even when each fitted library could use its own best fixed choice among the same three groups, the simultaneous lower bounds on the gain in net utility were positive on both panels (+0.103 and +0.047 points).

Existing interactive retrieval methods tell us how to ask questions and use the answers. Our contribution is a way to test a model choice those systems depend on. Our theoretical example shows how two systems can tie before any question and with the search intent fully known, yet favor different questions. In a separate SUN follow-up, we measured how including feedback changed the value of the same ranker assignment. Testing the same group assignment before any answer understated its advantage over the full set with paid feedback by 0.119 [0.083, 0.159] and 0.094 [0.050, 0.141] points. This gives a designer a concrete reason to evaluate a proposed group of rankers with the feedback policy it will actually use.

These follow-up comparisons use previously evaluated SUN panels and the benchmark’s attribute-vote response model at an assumed cost of 0.5 AP points per answer. The adjusted bounds resample classes with five trained libraries held fixed. Human response costs remain unmeasured.

Reviewer_tXZNOpenReview ↗

Thanks for the rebuttal

Thanks for the rebuttal and the new comparisons. I believe the paper would need significant reframing throughout and more thorough evaluations for it to be fully ready for this venue.

Reviewer_XJ24OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that two retrieval systems can have the same blind and fully condition-known endpoint scores and still differ in how much intermediate feedback is worth. It gives an exact average-precision (AP) counterexample, defines endpoint-preserving action deletion and the deletion risk for a given set of probes, and studies feedback-aware replanning on UT-Zappos and CUB. The experiments then ask three separate questions: whether deletion can hurt at all, whether it hurts on the question paths a good policy actually uses, and whether a deployed policy improves.

Critical strengths

The theoretical construction is clean. In Table 2 (p. 4), two systems have the same blind AP (239/360) and the same known-condition AP (1), yet prefer opposite questions. Equations (2)–(5) (p. 3) then connect deletion risk to policy loss. The experiments are also fairly thorough. Besides the full census of 1,920 cells, it checks sensitivity to tariff and noise, a separate transport cohort, fusion and learned actors. The one clearly positive result, two-answer calibration for a single ranker, is reported without overselling (Table 15, p. 23).

Critical weaknesses

The natural-image evidence shows that reversals exist, but not that they matter in practice. In Appendix N, forced one-question reversals occur in only 13 of 1,920 cells, and paid reversals in six (Table 23, p. 28). No query exhibits a reversal across all three seeds, and the worked example (Table 24) was selected after the census was observed. Endpoint-preserving deletion also preserves all optimal trees in 1,887 cells, including 318 of the 351 cells with positive allowed-probe risk (Table 7, p. 19). Under the primary tariff the mean replanning gaps are very small (Table 9, p. 20). So the diagnostic is correct, but I do not see evidence that it would lead to a useful intervention.

Operational value is not shown. On the transport cohort, the uncertainty menu does not beat the strong endpoint menu (Table 4, p. 7). Two-answer calibration improves single-ranker AP by only about 0.0025–0.0034, and fusion gives no gain or a loss (Table 15, p. 23). Several evaluations use true AP or oracle response laws, which are not available at deployment. The simulated tariff also leaves out annotation, fitting, storage, fusion and serving costs (Sections 3 and 8). The related work should also discuss WhittleSearch (Kovashka et al., CVPR 2012) and dialog-based interactive image retrieval (Guo et al., NeurIPS 2018).

Questions for authors
  1. Can a selector that is fixed before evaluation and uses only inputs available at deployment find held-out cases with reproducible net gains over the strongest endpoint and posterior controls?
  2. How would the conclusions change with measured human response costs and with payoff and response models that are fully estimated?
  3. Why are classic attribute- or dialog-feedback retrieval methods not included as baselines under the same menus and budgets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XJ24

Can a fixed policy using public inputs beat strong controls?

With the CLIP retrieval library fixed, replacing the DINO observer with CLIP improves two-answer AP by 0.67 points [0.16, 1.18] (adjusted 95% interval). Both receive the same twenty-eight candidates, including posterior planning, myopic payoff, entropy and fixed-question controls under single-ranker and Z-score ranking. Selection uses calibration queries (SOURCE) before evaluation (TARGET). All five observer-training runs have positive mean differences. We average three libraries within each of 206 shared query components, resample runs and components separately, and adjust within and across supplementary analyses.

Holding the original selected policy fixed retains 0.64 points; policy reselection contributes 0.03. This decomposes the 0.67-point total descriptively, without separate significance tests. The common answer cost cancels at exactly two answers (Appendix O.3).

With stopping allowed, DINO/CLIP AP is 0.5843/0.5907, mean answers 2.0000/1.9179 and utility 0.5743/0.5812 at synthetic cost 0.005 per answer (Table 6). The utility gain is 0.69 points [0.17, 1.20]; its lower bound is positive but below the 0.2-point practical margin. This supports observer substitution, not an independent stopping benefit. First-question regret training did not improve over matched squared-error training or strong controls. The same-observer two-answer entropy contrast has a zero-containing descriptive interval (Appendix O.3).

A separate SOURCE-calibrated policy was frozen before evaluating all 407 eligible queries against 2,121 gallery images, with stopping allowed at a two-answer cap. Against the SOURCE-selected single-ranker stopping control:

EncoderAP change (points)Mean-answer changeUtility change (points), 97.5% interval
DINO+0.122−0.220+0.232 [0.088, 0.375]
CLIP+0.073−0.344+0.245 [0.062, 0.430]

Utility is AP minus 0.005 per answer under the synthetic response law. Intervals adjust for both encoders and resample queries after averaging worlds and three fixed libraries. The single-ranker policy trails SOURCE-selected reciprocal-rank fusion by 0.49/0.90 utility points for DINO/CLIP. Calibrating planning with that terminal gives +0.15 points per encoder: the adjusted interval excludes zero for DINO but includes zero for CLIP. This is a calibration effect, not a new acquisition-objective gain.

What information does the policy use?

The response model uses TRAIN; the payoff predictor uses FIT and is tuned on TUNE. Policy selection and noise estimation use synthetic SOURCE data, not measured human reliability. Evaluation decisions use only the query image and purchased answers. True AP, hidden conditions and future realized answers are evaluator-only. Integrating possible answers under the fitted model does not reveal the future answer. Calibration data are separate; prior use of evaluation images and labels is documented.

Does execution reproduce the policy, and what does it cost?

Direct execution matches cached questions, answer-dependent choices and final rankings in all 1,440 calibration sessions across fifteen run–library combinations. Each observer has 480 warm sessions. DINO/CLIP median latency is 75.34/37.16 ms (95th percentiles 80.65/41.22 ms). CLIP shares retrieval features, reducing encoder calls from two to one.

Each session is one real query padded to 64; time is not divided by 64. Both encoders are resident. Separate residency tests load only required encoders: DINO still needs CLIP retrieval; CLIP shares one encoder. Peak allocated memory is 1.404/0.487 GiB for one selected pair; Appendix O.4 reports all six tests. These timings neither measure concurrent serving nor validate the older execution path.

We have not measured human response accuracy, response time, or subjective effort. A dedicated user study to estimate these quantities reliably was beyond what we could complete within the rebuttal period. The reported execution times measure computational cost, while our utility results remain conditional on the stated response-noise and per-answer cost assumptions.

The natural census retains thirteen reversals, six paid cases and no query reversing across all three seeds. Observer substitution is separate. Section 2 discusses WhittleSearch [1] and dialog retrieval [2]. These were not reproduced: relative-attribute and dialog answers differ from binary questions in information and human effort, so matching turns alone is insufficient.

References

[1] Adriana Kovashka, Devi Parikh, and Kristen Grauman. WhittleSearch: Image Search with Relative Attribute Feedback. CVPR, 2012.

[2] Xiaoxiao Guo, Hui Wu, Yu Cheng, Steven Rennie, Gerald Tesauro, and Rogerio Feris. Dialog-based Interactive Image Retrieval. Advances in Neural Information Processing Systems 31, 2018.

Dear Reviewer XJ24,

Our response clarifies the information available during execution and reports observer comparisons using a common retrieval library and policy menu, together with measured computational costs. Human response costs remain unmeasured. The observer comparison is separate from the endpoint-insufficiency diagnosis.

Do these clarifications address your concern about the information available at execution time? Separately, does our response clarify the practical implications and limits of the diagnosis, and what concern remains about its significance?

Thank you for your time.

Thank you for your response, but my main concern remains unresolved. I will maintain my score.

AuthorsOpenReview ↗

Expected answer use and utility from ranker selection

Dear Reviewer XJ24,

Your review asked whether the diagnosis can lead to a useful intervention using only information available at deployment. We tested a decision the system can make before the user answers anything. For each image, it chooses among three groups of trained rankers by predicting the quality of the final ranking under possible answers and subtracting their expected cost. The rule uses estimated answer probabilities and ranking quality. We saved its choices before joining them to evaluation outcomes. The endpoint selector had the same groups and downstream policy, with up to two questions. Its weighting of performance before feedback and with known intent was tuned per library on 128 separate validation queries.

We compared the expected number of answers each policy purchased across all queries, including early stops. Average precision (AP) and net utility differences are reported in points (100 times the raw difference).

Compared with the tuned endpoint selectorSUN 384SUN 128
Fewer expected answers per 1,000 queries107.5 [84.2, 132.2]79.1 [36.3, 127.3]
Change in average precision (points)+0.0034 [−0.0238, +0.0307]+0.0042 [−0.0298, +0.0384]
Change in net utility (points)+0.0571 [0.0280, 0.0883]+0.0437 [0.0090, 0.0853]

The intervals in the table resample classes with the five fitted libraries held fixed. The AP and answer-count intervals adjust jointly across four comparisons. The utility intervals adjust across twelve.

At 0.005 raw AP per answer, the reduction in answers contributed 0.0537 and 0.0395 utility points, respectively. It accounts for most of the observed gain. The AP intervals allow both a small loss and a small gain. The figures per 1,000 are expected counts under the benchmark’s attribute-vote response model. Converting them to time saved would require measuring how long people take to answer the particular questions each policy asks.

The comparison changes with the baseline. Against the validation winner among 18 posterior planners, net utility increased by 0.087 [0.012, 0.167] and 0.226 [0.059, 0.402] points. Against the winner among all 36 policies, the rule asked about 0.26 more questions per query on both panels, gained 0.234 and 0.338 AP points, and gained 0.105 and 0.209 utility points after paying for those answers. None of the adjusted utility lower bounds exceeded our 0.2-point practical reference. When we doubled the answer price, we kept our selector and its question paths fixed. The comparator reselected from the 36 policies at the new price led on SUN 384.

These are follow-up comparisons on previously studied panels. When we also resampled libraries, only the comparison with the 36-policy winner retained a positive utility lower bound on both panels. The original, more detailed feedback rule showed no clear additional gain over the simple rule when both could select a group per image.

Accounting for feedback changed an executable model choice and reduced expected answer use relative to endpoint selection. Whether the original deletion diagnostic identifies the individual queries that benefit remains open. Actual user time, accuracy, effort and end-to-end cost also remain unmeasured for this intervention. WhittleSearch and dialog retrieval elicit different kinds of feedback, so a comparison at matched human effort would require measuring the cost of their answers as well. We would welcome your assessment of this narrower operational result.

Reviewer_1uXGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Addressing the issue that equal endpoint scores do not guarantee equally valueless intermediate feedback, the authors construct a precise AP counterexample that reverses the optimal question while matching dimensionality, number of actions, and endpoint scores. This is verified through exhaustive census on natural image datasets (Zappos, CUB), revealing disjoint optimal single-question sets in 13 out of 1,920 cells. Building on this, they propose a three-tier diagnostic framework—diagnostic risk, policy relevance, and operational value—and apply an endpoint-preserving deletion method, which retains all optimal trees in 1,887 out of 1,920 cells. Results show that although the deletion diagnostic risk is positive, the replanning policy gap is small (mostly below the 0.002 threshold). Publicly predicted endpoint scores offer no replanning advantage over matched menus. With exactly two answers, SOURCE-calibrated planning improves single-ranker AP, but fusion gains have yet to be established.

Critical strengths

1、Strong originality of the evaluation perspective 2、Rigorous experimental control: the authors separate diagnostic risk, policy relevance, and the deployed policy into three layers, and use endpoint-preserving deletion, matched-menu baselines, own-STOP controls, and crossed response-noise and terminal-rule tests.

Critical weaknesses

1、not easy to understand 2、The core conclusions are mostly negative and existence-only, lacking a deployable positive gain: the paper repeatedly stresses that "this does not establish a deployable selector"; publicly predicted endpoint cores show no advantage under replanning relative to matched menus, and the main positive finding (single-ranker AP improvement) appears only in the narrow setting of exactly two answers, while fusion gains remain unestablished—leaving limited practical system value. 3、Limited evaluation scale and external validity: all evaluated populations are "historically exposed," introducing selection bias; sample sizes are small (e.g., only 128 READOUT queries, 256 CUB queries), and the three seed libraries share the same queries rather than being independent samples. The authors themselves acknowledge that zero-containing intervals do not constitute an equivalence result, and many key contrasts have intervals crossing zero.

Questions for authors

Can the single-ranker AP improvement observed at exactly two answers remain stable under wider question budgets (e.g., cap three or more) and under the fusion terminal? Could the authors provide a positive case for a fusion-aware acquisition policy, rather than only reporting that it is unestablished?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1uXG

Does the single-ranker improvement extend beyond two answers?

Yes, it extends to three answers. At exactly three answers, the planner improves AP over controls selected on separate calibration queries by 0.35 points with DINO and 0.47 with CLIP. The simultaneous 95% intervals are [0.10, 0.60] and [0.16, 0.78]. With stopping allowed before three answers, CLIP single-ranker utility improves by 0.47 points [0.14, 0.80]. Section 6 and Appendix O.2 report these results alongside the original two-answer comparison.

The intervals resample query components with the fitted libraries fixed and adjust across ninety nonconstant comparisons covering question budget, ranking rule, and stopping mode. The full analysis contains 108 comparisons; eighteen question-count differences are constant. Separately, the original question-set analysis retains the 6,144 conditions where only two questions are available and exactly three is structurally impossible.

Is there a positive result for fusion-aware question selection?

In the cap-three comparisons, fusion-aware planning remains inconclusive. With early stopping allowed, utility differences for CLIP are 0.21 points for reciprocal-rank fusion and 0.16 for Z-score fusion; both adjusted intervals include zero. Training the question selector for fusion has not established superiority over strong controls.

In a separate 407-query evaluation with stopping allowed at a two-answer cap, calibrated reciprocal-rank-fusion planning improves utility by 0.15 points for each encoder over the SOURCE-selected RRF control; the adjusted interval excludes zero for DINO and includes zero for CLIP. This is a calibration effect under that terminal and budget, not a gain from a new acquisition-training objective.

A separate experiment changes the model that interprets feedback while holding the CLIP retrieval library fixed. Under a fixed entropy-questioning and Z-score-ranking policy, replacing the DINO observer with the CLIP observer improves AP by 0.63 points [0.25, 1.03]. This interval holds fitted models fixed, resamples query components, and adjusts for four comparisons.

In the descriptive component analysis, the clearer gain comes from using feedback in the final ranking: that contrast is 0.37 AP points [0.23, 0.51], whereas changing question selection alone gives 0.15 [−0.10, 0.41]. These 95% intervals resample training runs and query components separately without multiple-comparison adjustment. Appendix O.3 separates this analysis from the adjusted system comparisons.

Giving both observers the same twenty-eight candidate policies and selecting on calibration queries yields a two-answer AP improvement of 0.67 points [0.16, 1.18]. Allowing stopping yields a utility improvement of 0.69 points [0.17, 1.20] under the declared synthetic cost. These supplementary intervals adjust within and across analyses. Their lower bounds exceed zero but not the 0.2-point practical margin. The comparison supports observer substitution, not a separate benefit of early stopping or the acquisition-training objective.

On Shapes3D, training the feedback head with balanced relation supervision improves AP by 0.34 points [0.12, 0.64] over category supervision under fixed entropy questioning. This gain is limited to observer training on a synthetic domain under the fixed policy; comparisons against the strongest calibration-selected policies remain inconclusive (Appendix O.5).

How are sample size and repeated seeds handled?

Five observer-training runs and three retrieval libraries share 206 query components. We average libraries within components and resample training runs and components separately; these are not 3,090 independent queries. Policies are selected on calibration data (SOURCE) before evaluation (TARGET), while the evaluation population retains its documented prior use. The revised abstract and Sections 1–3 and 6–8 introduce the ranker and observer decisions before the formal definitions. The original rare reversals, unsuccessful public-core controls, and negative fusion results remain.

Dear Reviewer 1uXG,

Our response extends single-ranker evaluation to exactly three answers: +0.35 AP points for DINO and +0.47 for CLIP over calibration-selected controls. Separately, early stopping under a three-answer cap gives CLIP +0.47 utility points [0.14, 0.80]. Cap-three fusion remains inconclusive.

Does this address the exactly-two-answer restriction for the tested single-ranker settings? If a concern about the budget or stopping comparison remains, could you specify it?

Thank you for your time.

AuthorsOpenReview ↗

Choosing the third question with reciprocal rank fusion held fixed

Dear Reviewer 1uXG,

Our earlier reply reported single-ranker gains with a three-answer budget. To address your question about fusion, we have now isolated the value of choosing the third question when several rankings are combined.

On two Zappos panels, we held the observer, the first two question–answer histories, and the final reciprocal rank fusion (RRF) rule fixed. Whenever the policy bought a third answer, we compared choosing the question with the highest predicted average precision (AP) under fusion against following a fixed question priority selected per library on 128 validation queries. These two policies purchased exactly the same number of answers. A separate control (STOP2) ended those same paths after the second answer and retained any earlier stops. It used the original choices for the first two questions. All eligible queries, including early stops, contributed to utility at a cost of 0.005 raw AP per answer.

Zappos panelGain over fixed priority (utility points)Gain over stopping at two (utility points)
206 queries, DINO+0.211 [0.101, 0.318]+0.273 [0.140, 0.428]
206 queries, CLIP+0.549 [0.366, 0.753]+0.619 [0.408, 0.854]
407 queries, DINO+0.264 [0.181, 0.354]+0.319 [0.224, 0.423]
407 queries, CLIP+0.408 [0.285, 0.540]+0.531 [0.393, 0.683]

All four settings gained from choosing which question to ask. In all four, buying the third answer also repaid its cost on average relative to STOP2 on the same paths. Three of the four adjusted lower bounds for this second comparison exceeded the 0.2-point practical reference. The intervals adjust jointly across 24 comparisons and resample shared query components with three fitted libraries per encoder held fixed. Both panels were previously studied.

Against an existing adaptive RRF policy, the incremental gain was smaller (0.034–0.083 points), and only the 407-query DINO interval excluded zero. A one-step rule that maximizes expected fused AP reproduced the chosen question in all 4,571 histories where the third answer was purchased. Under the stated response model and answer price, choosing the third question improved this RRF policy. In a separate full-system comparison, the policy selected on calibration data from a catalog of 33 policies spanning different final ranking rules outperformed the tested RRF system in all four settings. The third-step experiment leaves open what would happen if we jointly reoptimized the earlier questions and stopping decisions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We have uploaded a revised manuscript incorporating the additional analyses and clarifications described in our responses. References to newly added sections, tables, and appendices refer to this revision. We will consolidate further changes arising from the discussion into a subsequent update and clearly summarize those changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.