- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether a model’s ability to provide plausible-sounding factual answers is the same as its ability to determine whether evidence for those facts exists within a specific corpus. The paper constructed the "Factual Exposure Benchmark" (FEB) by exhaustively searching approximately 6.4 million Wikipedia articles to record the co-occurrence of subjects and objects. The authors then compared "positive" samples, where the subject and object appear together against two types of control samples: those with matched occurrence frequencies and those with altered objects. A classifier integrating probe features from six different models yielded an AUC of 0.5583 for the first condition and 0.9230 for the second. Paper also conducted further analysis based on whether specific tokens appeared in the training data for the Pythia model. The primary conclusion is that the evaluation results for these probes vary significantly depending on how the samples and labels are constructed.
Critical strengths
-
The design allows for the verification of labels at the unit level. It specifies the corpus used for exhaustive searching, the conditions for occurrence, the supporting context, and frequency information. This can enable other researchers to check the results using the same labels and control groups, which is beneficial for the research community rather than just reporting the detection performance alone.
-
Instead of interpreting an initially high AUC as proof of successful exposure detection, this work runs various controlled experiments for verification. This includes not only frequency matching but also the application of a classifier, based on the same 60 features to the labels present in the Pythia training data. The study also include data partitioning through entity and uncertainty estimation; specifically, it doesn't overclaim that low AUC equates to inherent undetectability.
Critical weaknesses
-
The experimental conditions were not sufficiently controlled to isolate the observed difference in AUC as being solely due to the effect of negative samples. Specifically, there is no indication that the set of positive facts remained consistent across the two conditions, which utilized 3,039 pairs for the "object-modified" condition and 878 pairs for the "frequency matched" condition. Consequently, the difference in AUC (0.9230 vs. 0.5583) could be influenced not only by the negative samples but also by the inherent difficulty of the positive facts and the number of training samples used. While the sensitivity to the choice of evaluation set is evident, the explanation provided in Section 4.4 for these results requires a more direct, controlled comparison.
-
Furthermore, the study fails to quantitatively demonstrate how the specific characteristics of the appearing labels influence the results. By definition, the approach does not examine the semantic nature of the relationship but instead relies solely on the co-occurrence of two strings. In some instances, such as the example "Pennsylvania Route 696 / Pennsylvania", the object is contained within the subject. If such matches are permitted, a simple mention of the subject could suffice to classify a pair as a positive instance. While the authors appropriately acknowledge the distinction between semantic grounding and mere string co-occurrence, a more precise assessment of the benchmark’s informational value would require reporting the prevalence of such cases and providing results excluding them.
Questions for authors
-
If we keep the same number of training samples (878 positive instances), features, and data-splitting procedures, but change only the type of negative samples, do the observed differences in AUC persist? Please also provide instance-level confidence intervals for the differences found in this comparison.
-
Are cases where the object is contained within the subject, or where the spans of the two entities overlap also classified as positive? I would like to know the proportion of such cases and see the results when they are excluded.
-
In Table 5, to what extent can the decline in performance after matching be distinguished from the effect of a reduction in sample size? Comparing the results against a randomly selected subset of the same size for each relationship type would help clarify the specific impact of the matching process itself.