- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Previous-Token Prediction (PTP) for prompt reconstruction. First training an inverse language model from scratch on token-reversed sequences generated through vocabulary-level probing of a target LLM, followed by fine-tuning on a small prompt–response dataset. The method reconstructs prompts from individual responses and evaluates both prompt recovery and response consistency through forward-model re-querying.
Critical strengths
-
The method is conceptually simple and implementable using standard autoregressive training, without requiring access to the target model’s weights, logits, or hidden representations.
-
Target-model-generated synthetic pretraining reduces reliance on external prompt–response datasets, with the main setting using only 400 external prompts for subsequent fine-tuning.
-
Evaluating regenerated responses alongside recovered prompts is valuable because it examines whether reconstruction preserves the target model’s behavior beyond surface-level prompt similarity.
-
The reported results show substantial improvements over the evaluated baselines on lexical reconstruction metrics in the main setting, while additional experiments explore transferability and the importance of synthetic pretraining.
Critical weaknesses
-
The central exact-recovery claim is difficult to assess. Appendix C defines Exact Match after truncating predictions to the ground-truth prompt length, which does not measure full-sequence recovery. Moreover, Figure 4 contains intermediate “Exact Match” scores, requiring clarification of how this metric is computed and aggregated.
-
The novelty relative to Pfau et al. (2023), which already uses reverse language modeling to generate prefixes conditioned on target suffixes, is insufficiently established. The paper cites this closely related work but provides neither a detailed distinction nor a direct experimental comparison.
-
The ablations do not adequately isolate the benefit of reverse-token modeling from that of substantial synthetic pretraining. Comparing synthetic pretraining against training on only 400 examples does not establish whether PTP outperforms a forward-order alternative under matched data and computational budgets.
Questions for authors
-
Could you provide the Exact Match implementation, explain the intermediate values in Figure 4, and report full-sequence EM without ground-truth-length truncation? Please also specify what the reported error bars represent.
-
What are the substantive methodological differences from Pfau et al. (2023), beyond the synthetic-data construction and application setting? Could you provide a controlled comparison with that reverse-LM approach?
-
Does PTP retain its advantage over forward-order synthetic pretraining followed by response-to-prompt fine-tuning when backbone, initialization, training data, and computational budget are matched?
-
How is prompt generation terminated without knowing the original prompt length, and how much does fine-tuning improve free-running prompt reconstruction rather than teacher-forced validation accuracy?
2 comments
Dear Reviewer kSAn,
As the authors' response deadline is fast approaching, please read through their rebuttal and let them know whether your concerns have been addressed or further clarification is still required.
Thank you.
AC
Gentle Reminder
Dear Reviewer,
As the discussion period is nearing its end, we would greatly appreciate it if you could acknowledge our response. Please let us know whether we have adequately addressed your concerns and if there is anything that remains unclear or requires further clarification.
We would be happy to provide any additional details that may be helpful. We look forward to hearing from you.
Thank you for your time.