- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies the problem of deciding whether to keep, compress, or drop failed turns in a trajectory, to save cost and improve subsequent performances. It is motivated by the fact that failed turns can become obsolete once the model gains new context. The paper proposes a new framework, RECAP, that proves effective across models and metrics on processing failed turns.
Critical strengths
- The paper studies a novel yet practical problem: saving context not by brute compacting / truncating, but by carefully processing failed turns.
- The paper presents a strong framework: RECAP is competitive across 3 models on 3 metrics -- success, token cost, and flip from success to fail.
- The paper provides sufficient code and appendix information for reproducibility.
Critical weaknesses
- The paper's introduction and abstract are hard to read. Beyond the first two paragraphs, the experimental meaning of KEEP, COMPRESS, and DROP are not defined anywhere in the intro, and the figure 1 is hard to read to draw the said conclusions. In addition, the enumeration of contributions in the end is also vague.
- Section 3 and 4 contain an excessive amount of math whose only role is to formalize, not to prove, anything. As a result, they are not central to the narrative, but further degrade readability.
Questions for authors
- Can the authors clarify the contribution of the work in the intro?
- Can the authors provide in natural language a succinct explanation of how RECAP works? -- As I understand it, this is not an optimization paper, so the formalism seems a little unjustifiable.
Response to Reviewer 9scZ
Thank you. We agree with both presentation concerns and have rewritten the corresponding parts.
Abstract, introduction, Figure 1.
Sections 3 and 4. The main text keeps two displayed formulas (the measured effect of an edit and the decision rule) and the statement of the guarantee; the rest moves to the appendix. The formalism serves one purpose: to state exactly what the guarantee covers. It is not an optimization result.
How RECAP works, in plain language.
Contributions, restated.
One correction to a listed strength. Your review lists RECAP's token cost as a strength. After re-analysis with task-level intervals, its saving is distinguishable from zero on one of five backbones, and a new online experiment finds no difference from KEEP. The revision presents RECAP as a way to avoid harmful forgetting, not as an efficiency method.