- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In some settings (i.e. contracts, technical docs, online communities), there are alternative definitions of various terms that the authors call "local meanings." A model answering questions inside this setting may be unfamiliar with these "local meanings," and thus produce plausibly correct responses that leverage the term's original meaning (i.e. the layman's definition of "trademark" vs a specific Trademark discussed in a contract). The authors call these errors "silent misreadings," and show that these silent misreadings occur across model families and corpuses (both synthetic and real). The authors perform subsequent analysis and show that models' internal entropy cannot detect these errors, but small models can detect them with privileged access to the answers. Finally, the authors show that augmenting the queries with the correct local definitions will recover the errors.
Critical strengths
The paper is written is a very clear and concise manner. All metrics, models, and datasets were clearly described, and all experiments were naturally motivated.
Critical weaknesses
Overall, the aims of this paper are quite unclear. The authors seem to be portraying the reduced performance of the model when withheld critical information as a shortcoming of the model. Moreover, many of these results are in domains where the "local definitions" are real words with similar meaning. Overall, while the experimental setups and subsequent findings of this paper are sound, I believe it contributes no real research value other than identifying a potential class of error without any means to predict it.
Major:
- Your first finding is that models queried without the definition text perform worse than models queried with the definition text. This seems less of a finding on model shortcomings and more of a finding on the necessity of information?
- The risk score is not just defined with respect to the query, but also the ground truth answer. I imagined the risk score as a way to diagnose whether an arbitrary question could contain missing definitions when you wrote . However, implicit in this definition is the log-probs of the ground truth answer text. As such, I'm confused as to the usefulness of this Risk score and the overall findings in Section 7. If the only way we can use the Risk score is when we have the gold answer, why do we need a proxy model to find the errors of the deployed model when we could simply check the gold answer? Since it needs the gold answer to compute, it cannot be used in settings where the ground truth is not available.
- The best patch to these issues is just to naively include the text in the definition as per section 8. This does not feel like a particularly novel finding.
Minor:
- On your second finding, there is a storied literature on how LLMs responses are rarely calibrated with respect to their logits:
- How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering
- Language Model Probabilities are Not Calibrated in Numeric Contexts
- Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
- All your experiments are constrained to multiple choice diagnostic questions written by an unknown model [160]. Would your results generalize to the full datasets with text?
Questions for authors
Please clarify if I have misunderstood anything written in the manuscript, especially w.r.t the Risk scores. Overall, I question your second implication that "Knowing that a definition is needed is a distinct capability." In settings where the relevant definition is not in context or even accessible to the model, how would you expect that the model should know there is an alternative definition it should be using. While this may make sense in settings where the terms are OOD words that the model tries to associate with some meaning, the examples that you showed in your paper seem to indicate that these terms are normal words (i.e. Trademark, Products, Term).