- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether current Language Model (LM) continual learning evaluations confound the model's knowledge and evaluation protocol, such as response format. In the short-answer setting, the experimental results show that fine-tuning reduces computation accuracy more than retrieval accuracy. The results show that LM evaluation is sensitive to the target format and the evaluation protocol.
Critical strengths
- The experiment findings might be robust as experiments were conducted across three random seeds.
Critical weaknesses
- The paper is not self-contained. Several terms and experimental details are not included or discussed. Without these details, the experimental results, their implications, and conclusions are unclear to me. For example:
- There is no formal definition of computation and retrieval accuracy. While the paper defines the evaluation framework in Section 2, it does not clearly discuss the meaning and details of these two accuracies.
- The paper states that the experiments are conducted on 14 tasks. However, it does not say which tasks are tested in the main text. Without mentioning the datasets, the meaning of the experiment numbers is unclear. Also, the “200-item suite” has not yet been introduced, despite being mentioned in many experimental results.
- Methods mentioned in the experimental results are not introduced in the main text. For instance, what are the "rational-first training", "label-only," and "prompted" methods mentioned in Table 1 and Figure 2? The main text does not clearly define these methods.
- If I understand correctly, this paper argues that the performance decline after fine-tuning reflects both model knowledge forgetting and response-format shift. While this is a known phenomenon (related works cited around Line 90), this paper is basically measuring this phenomenon in one specific scenario: the "short-answer setting". Meanwhile, this paper does not propose a method to mitigate or solve this problem (or, even if the paper does, the current presentation is unclear to me). Therefore, the contribution of this paper is limited.
Questions for authors
My questions are listed in Critical Weakness.