- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors examine three prompt adaptation methods on a text classification task, assigning civility labels (e.g., “insult” or “threat”) to online comments (Appendix D). The prompt adaptation methods are GEPA, prompt tuning, and prefix tuning, which differ by “insertion site,” e.g., GEPA optimizes hard prompts and prompt tuning optimizes soft prompts. The paper delineates several observations that distinguish the methods’ impacts on select models on the task, such as differences in the assignment of experts in MoE models (Section 4).
Critical strengths
The two research questions the authors pose (on pages 3 and 7) are interesting. I believe these questions are worth exploring, and I would be happy to read an ICLR paper investigating them.
Critical weaknesses
-
In my view, the paper's severe readability issues warrant a careful revision, without which the work is difficult to consume, evaluate, or reproduce. For instance, several terms and/or variables are non-standard, undefined, or ambiguous, and some details vary throughout the paper for reasons unclear. Additionally, there are confusing references to an earlier workshop draft (Appendix D, Appendix E.4, Appendix E.7) that include unexplained differences in settings.
-
It seems to me that the first of the two research questions (Section 3) remains unanswered, and the experimental design does not seem adequate to address such inquiry. While, indeed, the three methods differ by insertion site, the results do not appear to causally attribute downstream effects to this distinction.
-
The results of Section 3 are performed on only one small model, namely Gemma-2 2B. This is not a deal-breaker, but it limits the generalizability of the findings.
Questions for authors
-
Can the authors clarify the details of the dataset being used? I believe CivilComments has ~2M rows, with ~100k in the test set, but Appendix E.1 describes the test set as 2000 rows. Please let me know if I am misunderstanding.
-
What is the authors’ intention with Section 3.2? It adds padding to the seed prompt to (1) rule out that the inclusion of generic context improves task performance and (2) rule out that task performance increases monotonically with distance from the seed prompt (Figure 7a), but I am not sure why a reader would expect either of these possibilities.