- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper attempts to demonstrate that standard benchmarks can be misleading. Showing that a system performs equally well on standard retrieval tests (whether tested "blind" or with known conditions) does not mean it gives equal value when asking users for clarification or feedback. The paper shows a mathematical example where two systems look identical on paper, but when you ask a clarifying question, the system that seemed worse actually becomes the better choice.
Critical strengths
None
Critical weaknesses
-- The paper is extremely difficult to read and follow and uses heavy jargon. It is not accessible to a general ML audience. I had to look up the meanings of various terms in almost every paragraph. Even the abstract was quite inaccessible.
-- The contribution is more of a statistical nature rather than core ML. There doesn't seem to be a direct machine learning use case or learning scenario involved. It is centred around statistical evaluation of a retrieval system, which could apply to any statistical evaluation of a system not specific to machine learning applications. The paper's relevance to ICLR is unclear.
Questions for authors
Could you provide use-cases directly relevant to AI/ML?
Could you explain the contribution and its significance over related works?
4 comments
Dear Reviewer tXZN,
Our response gives two ML decisions: which trained image rankers to retain when users provide feedback, and which learned model should interpret that feedback. Do these examples clarify the ML use case?
Separately, our evaluation question is whether equal scores before feedback and with complete information ensure equal value after partial feedback. Does this clarify the contribution beyond methods for acquiring and using feedback? If not, which part of that distinction needs a clearer explanation?
Thank you for your time.
Which trained rankers should an image search system use for each query?
Dear Reviewer tXZN,
You asked for a use case that a machine-learning practitioner would recognize. Suppose an image search system has eight trained rankers. For a new query image, should it use all eight, or would a group of four work better once the user can answer a question? We tested a rule that reads all eight rankers’ predictions and chooses the full set or one of two groups of four before the first answer. The system can then ask a question or stop.
Compared with always using the full set, choosing the group per image increased average precision by 0.140 and 0.175 points on the 384- and 128-query SUN panels and used 0.113 and 0.059 fewer expected answers per query, respectively. These averages include every query. Even when each fitted library could use its own best fixed choice among the same three groups, the simultaneous lower bounds on the gain in net utility were positive on both panels (+0.103 and +0.047 points).
Existing interactive retrieval methods tell us how to ask questions and use the answers. Our contribution is a way to test a model choice those systems depend on. Our theoretical example shows how two systems can tie before any question and with the search intent fully known, yet favor different questions. In a separate SUN follow-up, we measured how including feedback changed the value of the same ranker assignment. Testing the same group assignment before any answer understated its advantage over the full set with paid feedback by 0.119 [0.083, 0.159] and 0.094 [0.050, 0.141] points. This gives a designer a concrete reason to evaluate a proposed group of rankers with the feedback policy it will actually use.
These follow-up comparisons use previously evaluated SUN panels and the benchmark’s attribute-vote response model at an assumed cost of 0.5 AP points per answer. The adjusted bounds resample classes with five trained libraries held fixed. Human response costs remain unmeasured.
Thanks for the rebuttal
Thanks for the rebuttal and the new comparisons. I believe the paper would need significant reframing throughout and more thorough evaluations for it to be fully ready for this venue.
Response to Reviewer tXZN
What machine-learning decision does this affect?
Consider an interactive image-retrieval system with several trained rankers. Its designer must decide which rankers to retain before deploying the system with feedback. A ranker can be unnecessary before any answer and when the desired condition is fully known, yet become useful after only one answer. Removing it solely because the two endpoint scores are preserved can therefore remove an intermediate decision that feedback makes useful.
Our exact average-precision construction shows that endpoint scores do not determine partial-feedback value. Two systems have the same dimension, five actions, blind AP of 239/360, and fully informed AP of one, yet prefer opposite questions. The useful ranking appears after partial information, so it need not be best at either endpoint. This is a realizable ranking example, not only an abstract payoff table. It does not imply a paid advantage at every budget and cost; at the studied two-question cap and cost 0.005, that example's paid advantage is zero.
When does deleting a learned ranker matter?
The next question is whether removing an action damages a useful question path. In the natural-image study, deletion preserves every optimal question tree in 1,887 of 1,920 cells. In 318 of the 351 cells with positive allowed-probe risk, it still preserves every optimal question tree. A possible loss therefore does not by itself show that the ranker is needed by an optimal feedback policy. There are thirteen endpoint-matched question reversals and six paid cases. Endpoint preservation alone cannot justify deletion, and possible-path loss alone cannot justify restoring a ranker: the relevant check is whether useful question paths retain their value. Sections 3–5 and Appendix A give this criterion; the separate restricted-width theorem and its assumptions remain in Appendix C.
In a separate CUB check with the fitted actor frozen, we examine query–library cases where the best achievable feedback value is unchanged. Terminal-menu restriction causes a loss above 0.2 AP points in 16/765 such cases at 32 dimensions and 16/754 at 128 dimensions, with questions, answers, STOP decisions and purchase counts unchanged (256 queries × three libraries per width). Restriction improves mean utility in both widths, so this identifies failures of individual fitted-policy preservation rather than average harm. The core is evaluator-derived; the result motivates reevaluating the fitted actor after a menu change, not a claim of a deployable pruner (Appendix J).
What completed learning experiment adds to the diagnosis?
A separate experiment tests a concrete learned-component choice. With the CLIP retrieval library and candidate policy menu fixed, replacing the DINO feedback observer with the CLIP observer improves two-answer AP by 0.67 points [0.16, 1.18]. Policies are selected on separate calibration queries and use only the query image and answers already obtained. The adjusted 95% interval resamples observer-training runs and query components separately while accounting for multiple comparisons. The clearer component gain is in using feedback to rank images; changing question selection alone remains inconclusive.
This supports observer selection under feedback. It does not turn the rare endpoint-matched reversals into a common failure mode or establish a new acquisition-learning method. The comparisons retain the documented benchmark splits and synthetic response assumptions; real human costs were not measured.
Interactive-retrieval methods provide mechanisms for acquiring and using feedback. We study what evaluation establishes about the learned system: an AP-realizable endpoint counterexample, a criterion connecting ranker deletion to preservation of optimal question paths, and a controlled observer-selection experiment. The results distinguish endpoint accuracy, preservation of feedback decisions, and performance of the fitted system. Value-of-information planning and rank fusion are existing tools; Section 2 explains this distinction from relative-attribute and dialog-feedback methods. Sections 3–8, Table 6, and Appendices A, J, and O connect the results to these ML decisions.