- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for learning reference-based textual scoring rules that align with human or LLM judge scores while encouraging truthful reporting. It uses frozen language-model representations, learned linear projections, and two families of proper scoring rules. The authors establish approximate-properness bounds under representation errors and evaluate statistical alignment and robustness to predefined textual manipulations
Critical strengths
-
The paper addresses an interesting problem by connecting proper scoring rules from information elicitation with learned textual evaluation metrics.
-
The experiments include multiple datasets, manipulation tests, and useful ablations of scoring functions and representation regularization.
Critical weaknesses
-
Assumption 3.1 requires text embeddings to encode linear properties of the beliefs expressed in natural language. However, pretrained language-model representations may also capture linguistic style, wording, and other nonlinear factors, and it is unclear how closely they satisfy this assumption for real-world reports.
-
The main Brier and Log-Cosh implementations use different backbone models and representation processing. Their results therefore cannot directly establish the superiority of one scoring family over the other. Moreover, removing the role penalty increases alignment while producing 30/40 significant positive manipulation effects, suggesting that practical robustness depends substantially on representation regularization rather than properness alone.
Questions for authors
- Can the authors provide more direct evidence that the learned projected representations satisfy the linear-property assumption on real-world peer-grading or QA examples? In particular, how large are the projected representation errors that determine the approximate-properness guarantees in these settings?
2 comments
Thanks to the response. I assume that the first "W2" and "Q2" in the response refer to "W1" and "Q1", respectively. Most of my concerns have been addressed, although the validity of Assumption 3.1 in real-world settings remains uncertain. Overall, I will maintain my original score.
We thank the reviewer for the encouraging feedback!
W2 the same experiment pipeline for both scoring rules
We repeat the same experiment pipeline as the current LogCosh for Brier on the peer grading dataset. Brier is manipulable by 0/40 attacks. The alignment results are shown below:
We will add the full evaluation in a revised version of the paper.
W2 regularization and robustness
Our regularization selects meaningful dimensions in the representation such that the dimensions represent uncertainty for a scoring rule to be proper. Without regularization, scoring rules cannot meaningfully score beliefs and are not guaranteed to be robust. With regularization, log-cosh is manipulable by 0/40 attacks; without regularization, log-cosh is manipulable by 30/40. We will add this comparison to the paper.
Q2 evaluation of the assumption for the full learning pipeline
It will be challenging to design an evaluation over a real review space. Our current test evaluates on a synthetic review space, using the trained log-cosh scoring rule. A real review induces a very high-dimensional semantic space and potentially correlated semantic meanings across dimensions. For such a space, it would be hard to design tests that have linear interpolation representations. We believe the current synthetic test and the manipulation results reveal promising evidence for our approach.