- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the agreement between data-similarity and data-influence methods for tracing LLM outputs to training documents. The authors compare their document rankings and observe an asymmetric relationship. Based on this finding, they propose combining similarity-based retrieval with influence-based re-ranking. Experiments are conducted across six language models.
Critical strengths
-
The paper investigates a potentially interesting problem concerning data-similarity and data-influence for LLM output tracing.
-
The experiments cover multiple language models and influence estimators.
Critical weaknesses
-
The paper does not sufficiently explain why agreement between data-similarity and data-influence rankings provides meaningful insights into LLM behavior. The connection between the research question and its practical significance remains unclear.
-
It is unclear whether the sampled documents were included in the actual training corpora of the evaluated models, raising concerns about interpreting the influence scores as training-data attribution.
-The paper reports asymmetric ranking agreement but provides limited explanation of its underlying causes or implications. Such agreement does not necessarily indicate better attribution quality.
Questions for authors
-
What meaningful insights into LLM behavior can be derived from the observed ranking agreement?
-
Were the evaluated documents part of the actual training corpora?
-
What explains the observed asymmetry between the two rankings?
-
Can the authors directly evaluate attribution quality and computational efficiency rather than relying on proxy-based metrics?