- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies gradient estimation from comparisons, specifically, whether antipodal-query rankings can recover gradient directions faster than within-pair signs rankings. It identifies some form of information hierarchy within this model, prove a nearly optimal bound, and certify information gains. Validation on two datasets show some improvements over the baseline method ZO-RankSGD.
Critical strengths
The paper seems to provide thorough theoretical analysis and discussion. I could not check the correctness, see critical weaknesses below.
Critical weaknesses
The way the paper is written is very convoluted, with many new (and often ambiguous) terms introduced without sufficient definition or context. For example,
(L014) Shared pair offsets create an information hierarchy: Mirror retains within-pair signs, Reverse couples reversed cross-pair orders to cancel offsets, and Full enforces all ranking inequalities.
What are shared pair offsets, within-pair signs, and cross-pair orders? Mirror, reverse, and full are also yet to be defined.
(L018) When it is at most , ...
c is undefined.
(L051)
f, b, and g are undefined. Undefined variables also show in the equations in the analysis. For example, in page 3. Also, multiple interpretations of the optimization problem exist without further clarifications. For example, is it a design optimization (e.g., find x that maximizes an objective function f(x)), or a predictor optimization (e.g., find b and g that minimizes a loss function L(f(x) - f^{gt}(x)))?
(L072) Even noiseless orders need not exclude the alternative directions allowed by signs. Nor does an accurate estimate alone establish an information gain: other directions can remain consistent.
What does it mean for an "alternative direction" to be "allowed by signs", and "other directions can remain consistent"=?
On several occasions, sentences have the correct words and grammars, but serve unclear purposes in the text. For example,
(L116) A batch is chosen before its feedback arrives.
(L119) A round is one feedback synchronization, not a count of comparisons or annotation efforts.
(L269) Gradients, optima, and held-out losses are audit quantities.
(L283) Lines guide the eye.
(L323) MSE also depends on the analytic-center selector; the exact cycle witnesses do not.
(L358) Squared loss is exactly linearplus-even at every center.
There are also paragraphs with multiple short statements where the readers have to infer how the sentences tie together. For example,
(L143) Strict observations are relaxed to weak candidate inequalities. Observed ties impose both inequalities. Adjacent-order inequalitie suffice. Repeated physical pairs share one offset. A common additive offset changes no order, so one offset can be fixed. On one shell , rescaling gives Across shells retain physical coordinates and the same gradient. Mirror keeps within-pair signs; Reverse adds all signed relations from Equation (4).
These issues occur throughout the paper and make the paper very hard to understand, even with significant effort and multiple re-readings. As a result, I might have missed other important strengths and weaknesses.
Questions for authors
The paper requires a major revision in its presentation before I can properly assess its technical contributions.