- Rating
- 2
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies a design over the prompts and responses in preference learning. Previous works studies this design separately over the prompts and responses for which a quadratic upper bound in the feature dimension was known. This paper shows that this bound is essentially tight and proposes a joint design over the prompts and responses which attains linear scaling. They consider a signed axis family based coverage where they prove these results.
Critical strengths
This paper demonstrates a clear separation between the separate and joint design over prompts and responses by considering a signed axis family. The authors provide intuition on how the joint design might be useful to attain a tighter upper bound. Efficient algorithms to do joint -optimal design might be useful in the broader regime of preference learning.
The results appeared to be technically correct as much as I checked in the main paper. I did not read over the appendices carefully though.
Critical weaknesses
I divide the weakness of the paper into two parts.
Presentation issues
I think the presentation of the paper leaves much to be desired. The paper uses words like "linear scaling" and "signed axis family" in the third paragraph of the first page without defining them or providing any pointer to it. I had to read the entire paper to figure out what they actually meant. A similar problem remains in the abstract too where the authors use the word "quadratic upper bound" but it it is not clear on what this upper bound is on.
Similar clarity issues remain in the introduction section and the diagrams (Figure 1) as well. On the left figure it is quite unclear what the colors actually refer to. Do they refer to the prompt or the responses? If 1,2 and 3 refer to the prompt, then I guess each color refers to a distinct prompt. But it is used for the feature axis too. This makes it quite unclear what it actually means.
While I could probably guess it after reading section 3.3 (signed-axis-family) where the both the responses and prompts share a dimension axis [d], it is quite hard to guess it from the initial introduction section.
Technical points
The authors essentially show that this gap exists on a signed axis construction that they propose, the significance of this result is unclear to me. The motivation for signed axis family including the definition of is not very clear to me either.
Thus, showing the quadratic vs linear gap on such a family over prompts and responses does not seem very significant to me. The Bradley Terry model considered in section 5 uses this feature in equation (10) and showing results on this setting does not answer the previous question either.
It would be interesting to see if such feature modelling is useful by providing the right references or performing experiments on some real preference data.
Additionally computational feasibility is not at all discussed. Computing a joint optimal design on prompts and responses might be computationally hard unlike computing a G-optimal design on just one set of features which is known to give an approximation of 2 in polynomial time.
Questions for authors
I would like to see a response to some of the technical points mentioned above in "Critical Weakness" section.