- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- The problem is well-defined, and the “datastore-free traceability” framework is clearly articulated.
- The attention mask ensures that the next-token distribution remains completely unchanged—a clean design.
- The experiments cover a wide range of scenarios: 3 models, 3 domains, scale scaling, counterfactual retraining, TRAK as a reference, and results for 5 seeds.
Critical strengths
1.The problem is well-defined, and the “datastore-free traceability” framework is clearly articulated. 2.The attention mask ensures that the next-token distribution remains completely unchanged—a clean design. 3.The experiments cover a wide range of scenarios: 3 models, 3 domains, scale scaling, counterfactual retraining, TRAK as a reference, and results from 5 seeds.
Critical weaknesses
- Table 18 and Table 1 are two completely different methods, but their results are exactly the same; I think authors need to explain this siuation
- With the setting of 30 queries per group, the target PPL and non-target PPL for the “None” row are exactly the same, and both equal the overall test PPL; the PPL for the single query q214 also happens to equal the overall value of 1.25. It is virtually impossible for the PPL of a subset or even a single document to exactly match the overall average (checked by AI, and I manually checked it should be a mistake and authors need to explain it)
- Random deletion is a very weak control. Deleting similar documents retrieved by BM25 or SBERT is also likely to cause a sharp increase in PPL
- How is Top-K calculated? Why isn't the cumulative curve concave?
- kNN-LM is traceable because the retrieved samples are directly involved in generating predictions
- Hyperparameter tuning and layer selection were both performed on the test set. Table 9 explicitly states that K, rank, and layers were tuned using the Finance Top-1 accuracy. Changing K from 2 to 3 resulted in a drop of 8.8 percentage points, indicating that the results are highly sensitive to hyperparameter tuning
- C.3. Table 9 and Table 17 both state that only five values of r ∈ {1, 8, 64, 128, 512} were sampled, but Figure 5 shows 10 data points (1, 2, 4, …, 512).
Questions for authors
see above
Gentle Reminder
Dear Reviewers and AC,
We sincerely thank you for your time and constructive feedback on our submission.
As the discussion period closes in three days, we would like to kindly follow up on our responses. We have replied to each review individually, and we would be grateful if you could let us know whether our responses address your concerns. If any questions remain, we would be glad to provide further clarification or additional results in the remaining time.
Thank you again for your effort and consideration.
Best regards,
Authors