- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper formalizes Rashomon Representation, a disagreement in representation across foundation models for an input. To predict whether an input can cause disagreement between models, the paper proposes a lightweight predictor trained with cross-model nearest-neighbor consistency over a reference dataset. Experiments are conducted to demonstrate the predictability of disagreement for test samples with the trained lightweight predictor. During inference, disagreement is predicted from a single representation.
Critical strengths
Using only a single representation and an anchor model, the proposed model is able to predict the disagreement between the representations.
Critical weaknesses
-
The problem is not well motivated. What’s the significance of predicting disagreement?
-
The experimental setup looks unsatisfactory. All the paper does is quantify correlation with their own created ground truth (neighborhood agreement). Is there another way to quantify ground truth?
-
The paper’s idea looks heuristic in the design of the models for predicting NA and the use of an anchor model. Is there any principled way to select those?
Questions for authors
see weakness section
We thank the reviewer for the valuable feedback and would like to clarify our core contribution. Our central finding is that cross-model representational disagreement can be predicted from a single model's representation. Neighborhood Agreement and the lightweight predictor provide a way to test this empirically.