- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
STRATA turns past terminal-task trajectories into knowledge and adds it to the prompt of a new task, without changing the model weights. Successful trajectories become multi-step procedures called Skills. Successful and failed trajectories from the same task are compared to produce Memories, which state when a rule applies and how to correct a mistake. Both kinds of knowledge are tagged with one of four capability layers, from command syntax to long-horizon planning. Before each new task, an auxiliary model picks the relevant entries. On Terminal-Bench 2.0, the paper compares STRATA with three experience-based methods (AWM, ExpeL, and ReasoningBank) and uses ablations to test the role of knowledge content, entry selection, and layer tags.
Critical strengths
Strength 1: STRATA gives a clear gain under a shared evaluation setup
In Table 2, STRATA raises the Pass Rate of DeepSeek-V4-Pro from 53.2% to 63.3%, above the best baseline, ReasoningBank (59.2%). The gain appears on both Medium and Hard tasks. The same library, built with DeepSeek-V4-Pro, also raises Qwen3-32B from 4.1% to 10.9% without fine-tuning. All methods share the same trajectories, the same model for building knowledge, the same scaffold and verifier, and the same prompt budget, so they are compared on equal terms.
Strength 2: The ablations show that how knowledge is organized and selected matters
In Table 3, keeping the knowledge but removing the layer tags lowers the Pass Rate to 57.3%. Keeping the library but switching to embedding retrieval lowers it to 50.6%. Generic CLI tips with similar format, topics, and length do not give the same gain. So the gain is not just from a longer prompt.
Critical weaknesses
Weakness 1: Paired extraction and distillation are not tested on their own
The paper never compares paired extraction with extracting lessons from successful and failed trajectories separately while keeping everything else fixed. The ReasoningBank comparison changes both how knowledge is extracted and how it is retrieved, so the two effects cannot be separated. The raw-trajectory ablation in Section 5.3 skips distillation, but it also picks trajectories by similarity instead of by an auxiliary model and leaves out failed trajectories. So its 4.9-point gap is not due to distillation alone. Memory-only reaches 61.8%, close to the full method's 63.3%, but this does not show how much the pairing step adds.
Weakness 2: The new contribution over AutoGuide is not clear
AutoGuide also compares trajectories with different outcomes on the same task, writes guidelines with conditions for when they apply, and picks rules based on the current execution history. STRATA cites AutoGuide but does not say which of its designs AutoGuide already has and which are new. The layer ablation already shows that capability layers help. The authors could start from this result and state which designs are new, such as the layers and the split into Skills and Memories.
Questions for authors
-
(Related to Weakness 1) Have you compared paired and non-paired extraction with everything else fixed? Section 5.3 credits the 4.9-point gap over raw trajectories to distillation, but that comparison also changes entry selection and leaves out failed trajectories. What evidence shows that the gap comes from distillation?
-
(Related to Weakness 2) Compared with AutoGuide, which new designs in STRATA matter most? Which existing results directly show that these designs help?
3 comments
2. Distinctive designs relative to AutoGuide and their empirical support
The two most important designs that distinguish STRATA from AutoGuide are its shared capability layers and its explicit treatment of procedures and contrastive lessons as complementary knowledge resources. Our completed repeated ablations support the value of the layers and of providing both knowledge types. Catalog-based selection supports their use for a new task.
AutoGuide already extracts conditional guidelines from contrasting same-task trajectories and selects guidance using the current execution history. It also learns from offline experience for transfer to non-overlapping test tasks. These are shared foundations. The distinctions concern the representation, organization and use of that knowledge, as summarized below. AutoGuide: Sections 3.1–3.3 and 4.1.3.
Table R2. Method differences relevant to the reviewer's question.
| Design | AutoGuide | STRATA |
|---|---|---|
| Knowledge representation | Context-conditioned guidelines derived from trajectory contrasts | Success-derived Skills encode action sequences and their purposes. Memories explicitly retain the failed action, successful alternative and applicability context. Sections 3.4 and 4.3–4.4. |
| Knowledge organization | A dictionary indexed by context, with matching of equivalent contexts | Both resources share L1–L4 capability tags, alongside a separate topic index. Tags enter catalog selection and prompt assembly. Section 3.3, Table 1 and Section 4.5. |
| Knowledge selection | Context matching and LLM guideline selection during execution | Before execution, an LLM selects categories, browses catalog summaries and reads candidate entries, separately for Skills and Memories. Section 4.5. |
Capability layers specify the decision level at which knowledge applies, from command syntax to long-horizon planning. Removing these tags lowers the three-run mean success rate by 2.62 percentage points, supporting their value beyond topic labels. Submitted paper: Section 3.3, Table 1 and Section 5.3, Table 3. Skills provide procedures distilled from successful trajectories, while Memories capture applicability conditions and corrective lessons from success–failure contrasts. Removing Memories or Skills lowers the three-run mean by 3.37 and 2.25 points, respectively, supporting their complementary value. Submitted paper: Section 3.4, Figure 2 and Sections 4.3–4.5. Before execution, catalog-based selection checks entries' roles, capability levels and content. Replacing it with embedding top-k over the same library lowers the mean by 9.36 points. This supports the procedure within STRATA. Submitted paper: Section 4.5 and Section 5.3, Table 3.
Table R3. Evidence for STRATA's organization, knowledge resources and selection. Success rate is the three-run mean ± sample SD on 89 tasks.
| Configuration | Design tested | Success rate ± SD | Δ, pp |
|---|---|---|---|
| Full STRATA | Complete configuration | 63.30% ± 3.61 pp | — |
| Flat | Remove capability-layer tags | 60.67% ± 4.90 pp | −2.62 |
| Skill-only | Remove Memory Pool | 59.93% ± 1.72 pp | −3.37 |
| Memory-only | Remove Skill Library | 61.05% ± 3.43 pp | −2.25 |
| Embedding top-k | Replace catalog-based selection | 53.93% ± 2.97 pp | −9.36 |
Overall, STRATA contributes a capability-layered organization of complementary procedures and contrastive lessons, together with a selector that reads this structure. The repeated ablations provide empirical support for retaining the layers, supplying both knowledge types and using catalog-based selection. These are the concrete designs and corresponding results that distinguish our contribution beyond the shared idea of learning conditional guidance from trajectory contrasts.
Thank you again for your feedback. We hope our responses have clearly addressed your questions and concerns.
Thank you for the detailed response and the additional experiments. The controlled comparison between paired and independent extraction in Table R1 clearly shows that pairing same-task outcomes yields a more compact and effective memory pool. The design comparison with AutoGuide in Table R2 and the repeated ablations in Table R3 also clarify the roles of capability layers, complementary knowledge types, and catalog-based selection. These results address my main questions, and I will keep my positive score.
Thanks for the insightful feedback! We will respond to your questions point by point below.
1. Effectiveness of success–failure paired memory distillation
Our additional experiments support the effectiveness of success–failure paired memory distillation. With STRATA's retrieval procedure, memory presentation and context budget unchanged, Paired achieves 63.30% average success versus 57.68% for Independent.
Both conditions draw from the same source trajectory pool and use a full-batch size of 24 trajectories. Each Paired batch contains 12 successful–failed pairs from the same tasks. Each full Independent batch contains either 24 successful trajectories or 24 failed trajectories. Thus, the two outcomes of a task never appear together in an Independent extraction input. The remaining retrieval, presentation and context-budget settings follow STRATA. Submitted paper: Sections 3.3–3.4, 4.4–4.5 and 5.1. After duplicate consolidation and support-frequency filtering, Independent retains 313 Memories, compared with 168 for Paired. Both conditions use their complete resulting memory pools. Table R1 reports the counts at each level, together with the three-run results.
Table R1. Memory extraction comparison. The layer columns give memory entry counts. SD is the sample standard deviation across three runs of 89 tasks.
LLM extraction and subsequent consolidation do not guarantee identical entry counts at every capability level. We therefore retain the naturally produced libraries without imposing layer quotas. The libraries have similar broad layer distributions, indicating similar coverage of capability levels. One possible explanation for the larger Independent library is that it summarizes successful practices and failure risks separately. Related experience can consequently survive consolidation as rules with different wording, types or contexts, potentially retaining more redundant formulations. Paired presents the two outcomes together and links the failed condition to a successful alternative in one rule, helping the extractor focus on the distinction that matters for the task.
Independent contains 86.3% more entries, yet its average success rate is 5.62 percentage points lower. Under the same context-budget specification, a larger library does not automatically provide more useful guidance to the agent. Same-task success–failure comparison gives Paired a direct reference for identifying the conditions and actions that distinguish the outcomes. The observed combination of a smaller library and higher average success is consistent with this advantage of paired extraction.
Overall, the additional experiments support the effectiveness of STRATA's success–failure paired memory distillation. Compared with independent extraction, it produces a smaller memory library while achieving a 5.62-point higher average success rate.