- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual prompt tuning when task IDs are unavailable for prompt selection. It combines representative training examples, label normalization and constrained decoding with gradient-based prompt selection and merging. Experiments cover classification sequences, SuperNI, and a math/tool benchmark. The method reduces prompt storage and improves several reported accuracy and retention measures, with some accuracy trade-offs.
Critical strengths
- The paper addresses a practical limitation of accumulating task prompts: inference must remain usable without task-specific prompt routing while prompt length stays manageable.
- Evaluation includes multiple task orders and architectures, plus open-ended generation. These experiments broaden the evidence beyond fixed-label classification.
- The appendices provide runtime measurements, threshold sensitivity, and Random/FIFO comparisons. The cumulative ablation helps distinguish the retention gains associated with decoding from the storage savings of compression.
Critical weaknesses
-
The test-time output-space information is unclear. Section 4.1 takes as input and constrains predictions to the corresponding . How is this candidate set obtained in a mixed task stream? A request-provided label set is a reasonable setting, but dataset-provided output spaces would supply information relevant to task identification. Baselines should receive the same label normalization and decoding constraints. Otherwise, the retention gains cannot be separated from output-space information and formatting corrections.
-
The retention metrics need reproducible definitions. Section 6.3 defines FTC as a task count, yet Tables 4-5 report 77-95 for sequences containing 15 tasks. If these are accumulated task-time events, the aggregation must be stated. The relationship between average accuracy and BWT is also unclear: Table 4 reports 75.70 accuracy and -0.7275 BWT for ProgPrompt on T5-large/L1. These cannot both represent final average accuracy and standard BWT on a [0,1] scale. Please provide the metric formulas, units, and full evaluation matrices.
-
Two algorithm definitions do not produce the stated behavior. The Section 4.1 formula leaves positive probability on forbidden tokens. Hard masking requires excluding their logits, and multi-token labels need sequence constraints. Also, Eq. (3) excludes scores equal to the threshold. With a single retained prompt and zero score variance, both sets are empty, so Algorithm 1 keeps only the new prompt. Please specify the actual masking, tie handling, and initialization or compression-trigger rules. These may be presentation errors, but the described algorithm is not sufficient to reproduce the system.
-
The gradient score has not been validated as a measure of historical redundancy. A small current-task gradient can indicate that a prompt is irrelevant to the new task while still essential to an old one. The reported prompt similarities and Table 25 are useful, but do not establish safe knowledge preservation. Tables 6 and 26 also give slightly better BWT without gradient selection, contrary to the statement that full GRID is best. A matched-budget comparison and historical-task losses immediately before and after merging would better test the proposed mechanism.
-
Several conclusions overstate the reported results. Section 6.1 claims approximately +3.8/+3.6 points on Flan-T5-base/large under negative transfer, whereas Table 21 gives +1.15/-0.74 points. GRID has lower average accuracy on four of that table's six backbones, despite its caption claiming consistent improvement. These discrepancies need correction. The paper averages three runs, but provides no dispersion for the small selection-strategy differences, which further limits the mechanism claims.
Questions for authors
See weaknesses