- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PGA inserts learned summary tokens at the punctuation boundaries, and limits attention to such summaries at a distance, while keeping full attention locally. Improves downstream accuracy and synthetic multi-token retrieval for 1B and 256M models. The paper also saves KV-cache by dropping older raw-token states.
Critical strengths
-
Table 1 makes a convincing case for combining summary tokens with the restricted attention mask. PGA reaches 96.5–99.96% across all four settings, while summary tokens alone fail on the harder cases.
-
The improvements are not confined to the synthetic task. PGA improves all seven benchmarks after continued pretraining at 1B and six of seven when training from scratch at 256M. Both scales also show lower training loss.
Critical weaknesses
-
The main distinction from Activation Beacon [1] appears to be punctuation-based placement, but the paper does not isolate its value. A fixed-interval control with the same summary budget and attention mask would be helpful here?
-
PGA gives every 16–128-token segment the same four summary tokens, then removes direct access to older raw tokens. This fixed capacity may be restrictive when exact details from several segments are needed. The synthetic task retrieves only one block, while Tables 2–3 do not vary evidence distance. A long-context test with evidence beyond the local window may help assess this trade-off.
[1] Peitian Zhang et al. Long Context Compression with Activation Beacon. ICLR, 2025.
Questions for authors
-
What local window and training sequence length were used, and what KV-cache reduction does that imply for the reported runs?
-
Does the 4M-token batch count original tokens or include the inserted summaries? How do training FLOPs and wall-clock time compare with the baseline?