- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TaskMatch, a lightweight relevance scorer for agent skill and procedural-memory retrieval. It learns from offline LLM-annotated relevance labels using frozen query and candidate embeddings, replacing or augmenting existing retrieval scores. Experiments report improvements in retrieval quality and downstream task performance on SkillsBench, ALFWorld, and TravelPlanner.
Critical strengths
- S1: The method is lightweight and reuses existing embeddings without additional online LLM calls, making it practical for repeated retrieval during agent execution.
Critical weaknesses
- W1: The paper offers limited methodological novelty. The claimed move “beyond semantic similarity” largely restates the established motivation for supervised relevance ranking, while the method introduces no distinctive design addressing agent harness-specific retrieval challenges.
- W2: The experiments insufficiently support the central claim of task-aware relevance beyond semantic similarity. Stronger embedding retrievers and established rerankers are not compared, and execution-related relevance errors are not systematically analyzed.
Questions for authors
- Q1: How is the reliability of the LLM-generated relevance labels validated, particularly for hard negatives and the required/supportive distinction?
- Q2: How sensitive is TaskMatch’s performance to the embedding model, its parameter scale, and embedding dimensionality? Have these factors been evaluated under the same retrieval setting?
10 comments
Response to Reviewer GL1S: Sensitivity to Embedding Models, Model Capacity, and Embedding Dimensions
Response to Reviewer GL1S: Sensitivity to Embedding Models, Model Capacity, and Embedding Dimensions
We have completed all three sensitivity studies under the same retrieval setting (identical frozen embeddings, graded relevance labels, and three random seeds).
(1) Model Capacity: Insensitive to Parameter Count — Model Capacity Is Not the Bottleneck
| Configuration | Parameters | Ret@1 | MRR |
|---|---|---|---|
| tiny | 148K | 70.59 ± 7.35 | 81.48 |
| Paper configuration | 298K | 73.53 ± 2.55 | 83.14 |
| wide_1m | 1.22M | 75.98 ± 0.85 | 83.92 |
| wide_5m | 3.52M | 75.98 ± 3.40 | 84.51 |
| wide_20m | 7.77M | 73.53 ± 3.89 | 82.97 |
As the parameter count varies from 148K to 7.77M (a 52× range), Ret@1 fluctuates only between 70.59 and 75.98, with no monotonic trend (it even drops back to 73.53 at 7.77M).
(2) Embedding Dimensions: TaskMatch Is More Robust Than Fixed Cosine, with Larger Advantages at Lower Dimensions
We applied a shared Gaussian random projection to both task and candidate embeddings (without using PCA), followed by L2 re-normalization, and recomputed the cosine baseline:
| Dimension | Cosine | TaskMatch | TaskMatch − Cosine |
|---|---|---|---|
| 128 | 38.24 | 65.69 | +27.45 |
| 256 | 42.65 | 71.08 | +28.43 |
| 512 | 50.00 | 73.04 | +23.04 |
| 1024 | 48.53 | 67.16 | +18.63 |
| 2048 | 52.94 | 71.08 | +18.14 |
| 3072 | 52.94 | 74.02 | +21.08 |
Two conclusions follow:
- TaskMatch is more tolerant of dimensionality reduction than cosine: from 128 to 3072 dimensions, cosine improves by +14.70 pp, whereas TaskMatch improves by only +8.33 pp (65.69 → 74.02).
- TaskMatch has a larger relative advantage at lower dimensions: +27.45 pp at 128 dimensions versus +21.08 pp at 3072 dimensions. This has practical significance: when deployment systems use smaller embeddings to save storage and bandwidth, TaskMatch offers even greater relative gains with lower integration costs.
(3) Embedding Models
We independently regenerated all task and skill representations using both embedding models (with exactly identical input texts: task_desc and skill documents), ensuring that the comparison is not affected by textual differences:
| Method | text-embedding-3-large (3072d) | text-embedding-3-small (1536d) | Δ |
|---|---|---|---|
| Cosine | 51.47 | 47.06 | −4.41 |
| TaskMatch | 71.08 | 67.16 | −3.92 |
| TaskMatch gain over cosine | +19.61 | +20.10 | +0.49 |
When switching to a smaller and cheaper embedding model, TaskMatch's absolute performance drops by 3.92 pp, but its gain over the host cosine baseline remains almost entirely unchanged (+19.61 → +20.10, a difference of only 0.49 pp).
In other words, TaskMatch's benefits are decoupled from the capacity of the embedding model: better embeddings improve all methods, while weaker embeddings reduce the performance of all methods, but the learned scorer's advantage over fixed cosine remains consistently present.
Therefore, TaskMatch's effectiveness does not depend on a specific embedding model, embedding dimension, or model-capacity configuration. None of these three variables changes the central conclusion that learned scoring outperforms fixed cosine similarity.
Thank you for answering my questions. My main concerns about methodological novelty (W1) and the evidence supporting the central claim (W2) remain unresolved, so I will keep my original score.
Response to Reviewer GL1S: Reliability of LLM-Generated Relevance Labels, Particularly Required/Supportive Distinctions and Hard Negatives
Reproducibility and Cross-Context Stability. Our 33 generic templates × 37 skills yield 1,221 judgments, generated in one run with 100% parsing success; grade 0/1/2 counts are 812/325/84. Templates exclude specific task entities, expert plans, and evaluation data.
phase_query× skill (≥8 pairs)For
find desklamp,alfworld-device-operatorreceives grade 1 in 55/55 occurrences, whilealfworld-goal-interpreterreceives grade 0 in 55/55.Inter-Annotator Agreement. A second model relabeled all 1,221 judgments using the same prompt:
Thus, none of the 84 reference grade-2 labels is assigned grade 0, and the precision of predicted grade 2 is 68/69 = 98.6%.
Functional Validity: Each Grade Carries Downstream Information. Although absolute label correctness is difficult to measure directly, we test whether each distinction is useful by removing it and measuring performance loss.
Required vs. Supportive. Binarizing grades 1 and 2 causes a substantial drop:
If the grade-1/2 distinction were mainly noise, merging them should dilute noise, not cause a 42.29-point collapse. This indicates non-redundant information about execution order: what is needed now versus what can be done later.
Hard Negatives. Under the same negative-sampling budget:
If hard-negative grade-0 labels were random noise, they should not systematically outperform random negatives. Their incompatibility is largely objective:
heat/coolare mutually exclusive state changes; single-/dual-object placement differs in cardinality;locate/verifydiffer in execution order; and forfind tissuebox,receptacle-finderandobject-retrieverare lexically similar but trigger different actions. Appendix E.1/E.2 provides the original rankings for independent verification.Scope of Claims and Mitigations. We do not claim that LLM labels equal expert ground truth or that individual labels are noise-free (the κ values above quantify disagreement). We claim that labels are (i) reproducible under fixed templates, (ii) stable across contexts, (iii) reliable in polarity (zero grade-2→grade-0 errors), and (iv) each level contributes measurable, non-redundant downstream information.
To address grade-0/1 boundary noise, the revision will: (1) apply stricter grade-1 criteria, requiring an explicit prerequisite/supportive or recovery role; otherwise, assign grade 0; and (2) clarify in the main text and Limitations that our method relies on group-level consistency and informativeness, not perfect individual-label accuracy.