acceptodds
Under review as a conference paper at ICLR 2027

TRAIL: LoRA as Traceable Parametric Memory with Trace Tokens

Abstract

Deploying large language models (LLMs) in high-stakes domains requires two long-competing capabilities: efficient domain-adaptive fine-tuning and traceability of each prediction back to its supporting training instance. Parametric LMs adapt efficiently but offer no per-instance index, while non-parametric kNN-LMs preserve instance-level traceability through retrieval but with substantial inference cost. Recent parametric imitations of kNN-LM remove the datastore but discard its traceability by construction. We propose TRAIL (Traceable Parametric Memory with LoRA and Trace Tokens), which recovers the instance-level traceability of kNN-LM within an efficient parametric module. On top of a frozen LoRA-adapted LM, TRAIL introduces a small set of learnable special tokens, called trace tokens, trained contrastively so that the hidden state of each prediction's trace token identifies its source training instance. On finance domain data, TRAIL attains 18–33% higher identification accuracy with 60–70% lower test perplexity compared to parametric memory baselines. This is achieved with 2× fewer trainable parameters, 60× smaller disk footprint, and 242× faster inference than kNN-LM. Counterfactual retraining shows that the identified documents are causally responsible, so TRAIL offers a foundation for accountable deployment of domain-adapted LLMs, supporting per-instance attribution, copyright compliance, and error diagnosis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Domain adaptation, Parameter Efficient Fine Tuning, Traceability, Training Data Attribution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5mvbOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  1. The problem is well-defined, and the “datastore-free traceability” framework is clearly articulated.
  2. The attention mask ensures that the next-token distribution remains completely unchanged—a clean design.
  3. The experiments cover a wide range of scenarios: 3 models, 3 domains, scale scaling, counterfactual retraining, TRAK as a reference, and results for 5 seeds.
Critical strengths

1.The problem is well-defined, and the “datastore-free traceability” framework is clearly articulated. 2.The attention mask ensures that the next-token distribution remains completely unchanged—a clean design. 3.The experiments cover a wide range of scenarios: 3 models, 3 domains, scale scaling, counterfactual retraining, TRAK as a reference, and results from 5 seeds.

Critical weaknesses
  1. Table 18 and Table 1 are two completely different methods, but their results are exactly the same; I think authors need to explain this siuation
  2. With the setting of 30 queries per group, the target PPL and non-target PPL for the “None” row are exactly the same, and both equal the overall test PPL; the PPL for the single query q214 also happens to equal the overall value of 1.25. It is virtually impossible for the PPL of a subset or even a single document to exactly match the overall average (checked by AI, and I manually checked it should be a mistake and authors need to explain it)
  3. Random deletion is a very weak control. Deleting similar documents retrieved by BM25 or SBERT is also likely to cause a sharp increase in PPL
  4. How is Top-K calculated? Why isn't the cumulative curve concave?
  5. kNN-LM is traceable because the retrieved samples are directly involved in generating predictions
  6. Hyperparameter tuning and layer selection were both performed on the test set. Table 9 explicitly states that K, rank, and layers were tuned using the Finance Top-1 accuracy. Changing K from 2 to 3 resulted in a drop of 8.8 percentage points, indicating that the results are highly sensitive to hyperparameter tuning
  7. C.3. Table 9 and Table 17 both state that only five values of r ∈ {1, 8, 64, 128, 512} were sampled, but Figure 5 shows 10 data points (1, 2, 4, …, 512).
Questions for authors

see above

Reviewer_8XMtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose a method which combines LoRa and special trace tokens to create a traceable parametric memory for kNN-LMs. They experiment their pipeline using different model families (Qwen, Llama, Gemma) for multiple domains. They also investigate a single domain for scaling Qwen model family. They also include causal analyses to validate their claims.

Critical strengths
  • Strength 1: Paper is well written, and the motivation is clear. Experimental design include many aspects from domain adaptation performances, causality analysis and scaling experiments. This breadth strengthens my assessment, since the method is validated along complementary axes rather than a single benchmark, making the central traceability claim more convincing.
Critical weaknesses
  • Weakness 1: For main models (Qwen2.5-0.5B, Llama3.2-1B, Gemma3-1B), trainable parameters with the LoRA config (rank: 512, alpha: 1024) are close to full fine-tuning. This undermines the paper's central efficiency claim since if the adapter's parameter count approaches that of full fine-tuning, the main advantage of the proposed method over a simpler full fine-tuning baseline is largely nullified, weakening the core motivation for the TRAIL design.
  • Weakness 2: Proposed framework includes appending trace token to each token in the sequence and memory consumption is not studied to highlight this trade-off.
Questions for authors
  • Can you provide a full finetuning (without LoRa) and LoRa with smaller rank (32 or 64) for a model from Table 1?
  • Can you provide inference time memory consumption of TRAIL against the baselines since the number of tokens at inference are doubled with trace tokens?
  • Can you also provide an analysis for general downstream (i.e. MMLU, ARC) performance change in addition perplexity?
Reviewer_wQfROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRAIL adds two learned trace tokens to a frozen LoRA-adapted language model. Their hidden states are used to predict which training document supports a text prediction, while an attention mask keeps ordinary language predictions unchanged. TRAIL agrees with nearest-neighbor source labels more often than the baselines and avoids a large retrieval datastore.

Critical strengths
  • Trace tokens improve document identification without changing the LoRA model's next-token probabilities. The model therefore gains a source readout without losing its existing language-model accuracy.

  • On Qwen2.5-0.5B and Finance 37K, TRAIL uses 521 MB of stored parameters versus a 31,744 MB kNN-LM datastore, with much lower per-token inference time.

  • Deleting TRAIL-selected documents before retraining increases perplexity on the targeted queries much more than deleting an equal-token random set. This is additional evidence beyond agreement with another attribution method.

Critical weaknesses
  • The main identification metric checks whether TRAIL selects the same document as kNN-1, an automatic nearest-neighbor reference. Agreement does not establish the true source, and deleting a whole document set cannot prove that every selected document independently caused the prediction.

  • Each domain has its own LoRA adapter and trace tokens. Results on Finance, Biomedical, and Law do not establish that a single tracing system works when those corpora are mixed or when test queries fall outside the adapted domain.

  • Inserting a trace token after every text token doubles the sequence length. The reported forward time is 0.043 ms/token versus 0.029 for LoRA, but actual generation throughput and peak memory are not reported. The classifier also grows with the number of training documents.

Questions for authors

See weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewers and AC,

We sincerely thank you for your time and constructive feedback on our submission.

As the discussion period closes in three days, we would like to kindly follow up on our responses. We have replied to each review individually, and we would be grateful if you could let us know whether our responses address your concerns. If any questions remain, we would be glad to provide further clarification or additional results in the remaining time.

Thank you again for your effort and consideration.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.