- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies errors in chaining tokenized procedural memories. TapMem augments TokMem with an explicit end-of-call token (ETS) and a transition-conditioned routing adapter (TCRA). The adapter adjusts memory-token logits at the initial selection and after generated call boundaries while retaining full-vocabulary autoregressive decoding. During memory learning, the backbone is frozen; the added embeddings and linear adapter are trained with autoregressive and routing losses. The evaluation covers APIGen compositional recall, SNI atomic recall, longer and dependency-structured trajectories, InterCode-Bash execution, and training-free initialization of unseen procedures. Results generally improve over TokMem, with larger gains in several unadapted and longer-chain settings.
Critical strengths
- On unadapted Llama 3.2 1B, Table 1 improves average tool-selection F1 from 82.4 to 88.8 and argument F1 from 67.0 to 74.0. Gains also appear on larger Llama models and the Qwen models in Table 5. Table 10 reports only 0.21M–0.41M trainable parameters for TapMem. These results support its practical use for improving frozen-backbone procedural memory without training many parameters.
- The explicit boundary mechanism is simple, and Algorithm 2 makes clear that inference uses generated boundaries rather than oracle boundaries. Tables 2/8 provide component ablations, while Figure 11 and Appendix E.5 report an increase in mean correct-prefix length from 2.78 to 3.43. Larger gains at longer call counts support the relevance of the intervention to chained recall.
Critical weaknesses
- The comparisons add a linear classifier and an auxiliary routing loss along with transition conditioning. Table 10 approximately doubles the trainable parameter count relative to TokMem, although the absolute budget remains small. Tables 2/8 establish that ETS helps and that the complete system improves further, but do not isolate added routing capacity and supervision as alternative explanations. A particularly relevant control is TokMem+DC, which already separates routing and generation embeddings; its original experiments do not settle this paper's compositional setting. A routing-capacity and supervision-matched control would help determine the distinct benefit of boundary conditioning. This matters to the recommendation because the main contribution is an explanation and remedy for a specific transition bottleneck, beyond the empirical gain from adding a stronger router.
- Regarding time cost, the paper reports no direct comparison of inference latency, throughput, or end-to-end execution time. Table 10 measures trainable parameters rather than runtime. Algorithm 2 introduces additional routing computation at the initial selection and after generated tokens; for the same call chain, each call's also adds an autoregressively generated token. The net time effect requires measurement, since changes in output length or fewer errors and retries could offset some overhead. Consequently, the time cost of the accuracy gains is undetermined, making the practical value of the small InterCode success gain particularly difficult to assess. This is not a claim that the method is slower; the concern is that the tradeoff between accuracy and time cost has not been quantified.
Questions for authors
- On a representative existing compositional setting, can you add a routing-capacity and supervision-matched control, such as TokMem+DC with matched routing supervision, and explain the additional benefit of explicit boundary conditioning? Separating initial-selection errors from subsequent-transition errors would make the comparison more informative. One focused control would suffice; I am not requesting a new benchmark suite.
- On one existing representative configuration, can you compare per-task latency and total generated tokens for TokMem and TapMem under the same hardware, precision, batch size, prompting protocol, and decoding budget, while also reporting accuracy? Please specify warm-up, timing boundaries, and repeated measurements. If using InterCode, distinguish model generation time, command or environment execution time, and end-to-end task time, retaining failed tasks to avoid bias from comparing only successful cases. One controlled comparison would support assessment of the time cost; a performance sweep across all models is unnecessary.
- Do the F1 metrics account for call order and repeated calls? Please specify the matching and averaging rules, and report an order-aware or dependency-valid trajectory measure on existing TaskBench/TRAJECT-Bench predictions, allowing equivalent valid orders where appropriate. Also clarify whether APIGen atomic examples are partitioned before composition and how TRAJECT-Bench source trajectories are partitioned. These clarifications establish the generalization scope; the current description alone does not demonstrate leakage.
5 comments
Official Comment [2/4]
\textcolor[rgb]{0.086,0.467,1}{\mathbf{W2}\text{\\&}\mathbf{Q2:}} Compare inference latency, throughput, end-to-end execution time, per-task latency and total generated tokens for TokMem and TapMem under the same hardware, precision, batch size, prompting protocol, and decoding budget, while also reporting accuracy. Please specify warm-up, timing boundaries, and repeated measurements.
A2: Thank you for your valuable question! We compare TokMem and TapMem using Llama-3.2-1B on APIGen, reporting per-task inference latency, total generated tokens, accuracy, throughput, and end-to-end prediction time. We also clarify the shared experimental settings and the details of warm-up, timing boundaries, and repeated measurements.
| Batch size | Method | Routing Acc (%) | Generation latency (s/batch) | Total generated tokens | Throughput (tasks/s) | End-to-end time (s) |
|---|---|---|---|---|---|---|
| 1 | TokMem | 69.74 | 1.97 | 35599 | 0.507 | 986.51 |
| 1 | TapMem | 83.82 | 2.00 | 35951 | 0.498 | 1003.55 |
| 256 | TokMem | 69.74 | 91.21 | 35599 | 2.724 | 183.57 |
| 256 | TapMem | 83.82 | 92.90 | 35951 | 2.674 | 186.95 |
Experimental Setup. We evaluate Llama-3.2-1B on the APIGen. Both methods run on one exclusively reserved NVIDIA A100-SXM4-80GB GPU. We measure inference time for both methods using FP32 precision. We compare matched batch sizes of 1 and 256, using greedy decoding with a maximum of 512 newly generated tokens. Routing accuracy is the number of correct tool predictions at matching positions divided by the total number of reference calls.
- Warm-up: Before measuring performance on the full test set, we run inference on two batches to reduce the impact of one-time initialization on timing. Warm-up is excluded from all reported metrics.
- Repeated measurements: Each configuration is evaluated three times using the same checkpoint and inference seed of 42. The table reports the means of these timing repetitions.
- Timing boundaries: Generation time is measured from when tokenized inputs are ready on the GPU until generation is complete, including prefill and decoding. End-to-end time covers processing the full test set, including input preparation, generation, and output parsing. Model loading, warm-up, accuracy scoring, and saving benchmark results are excluded.
- Prompting protocol: Following TokMem, both methods use user requests from the
user_inputfield of our APIGen compositional test set. We apply the same user-to-assistant template, without additional instructions or few-shot examples.
TapMem achieves higher routing accuracy at a small additional time cost in this setting. Accuracy increases from 69.74% for TokMem to 83.82% for TapMem, while per-task generation time changes from 1.97s to 2.00s. The time difference remains small at batch size 256, where processing the full test set takes 183.57s with TokMem and 186.95s with TapMem.
Official Comment [3/4]
\textcolor[rgb]{0.086,0.467,1}{\mathbf{Q3:}} Separating initial-selection errors from subsequent-transition errors would make the comparison more informative.
A3: Thank you for this suggestion! We group samples by initial-selection correctness to examine gains beyond the first choice, the results are shown below. We also assess sequential routing through correct-prefix length (Figure 11 and Appendix E.5).
| Method | First procedure correct | First procedure incorrect |
|---|---|---|
| TokMem | 97.4 / 80.9 | 53.9 / 41.0 |
| TapMem | 98.6 / 82.8 | 59.8 / 45.3 |
(1) Gains beyond the first choice. We analyze Llama-3.2-1B on the APIGen. We compute F1 over the complete call sequence within each group. Here each entry reports Tool Selection F1 / Argument Generation F1. These results demonstrate that TapMem improves procedure generation and mitigates the impact of initial-selection errors. The results are shown above.
| Number of procedures | TokMem | TapMem |
|---|---|---|
| 2 | 1.48 | 1.57 |
| 3 | 2.12 | 2.55 |
| 4 | 2.45 | 2.66 |
| 5 | 2.85 | 3.42 |
| 6 | 3.22 | 3.51 |
| 7 | 3.26 | 3.93 |
| 8 | 3.62 | 4.80 |
| 9 | 3.04 | 4.49 |
| 10 | 3.00 | 4.39 |
| Overall | 2.78 | 3.43 |
(2) Sequential routing reliability. We further evaluate correct prefix length, which measures the number of consecutive correctly selected procedures before the first routing error. As shown in Figure 11, TapMem increases the average correct prefix length from 2.78 to 3.43. The improvement is more pronounced for longer chains, averaging 1.34 over the three longest chain lengths versus 0.24 over the three shortest. These results further demonstrate TapMem's effectiveness in maintaining correct procedure transitions across longer sequences. The results are shown above.
Official Comment [1/4]
A1: Thank you for your comment! We conduct additional ablation experiments to verify the effectiveness of TapMem. The results on Llama-3.2-1B are presented below.
Additional ablations. We introduce three new variants:
Analysis. With the almost same parameter budget as TokMem, ETS-only improves both F1 scores by 1.6%, demonstrating the benefit of explicit procedure-boundary identification. With a parameter budget comparable to TapMem, both TokMem$^{\dagger}$ and TokMem$^{\ddagger}$ underperform TapMem, indicating that extra parameters or routing supervision alone cannot obtain the gains. Moreover, TapMem outperforms TCRA-only, despite sharing the same routing module and supervision. These results demonstrate the effectiveness of TapMem.