acceptodds
Under review as a conference paper at ICLR 2027

TaskMatch: Learning Task-Aware Relevance Beyond Semantic Similarity for Agent Harnesses

Abstract

Modern LLM agent harnesses augment the underlying model with reusable components such as external skills and procedural memories, making effective retrieval increasingly important as these repositories grow. However, existing retrieval pipelines often rank stored skills or memories by semantic similarity, which measures how similar a skill or memory appears to the current retrieval query but does not necessarily reflect how useful it is for completing the task. We introduce TaskMatch, a lightweight plug-and-play model that learns task-aware relevance from offline examples labeled as required, supportive, or irrelevant. Using the fixed query and skill and memory representations already available in the host retrieval pipeline, TaskMatch combines direct feature-wise matching with nonlinear interactions while leaving candidate generation and downstream agent execution unchanged. We instantiate the same architecture separately for two representative harness components—skill retrieval and procedural-memory retrieval—with fewer than 0.30M trainable parameters per model and no additional online LLM calls. Across both components, TaskMatch consistently improves retrieval ranking quality and downstream task execution. It raises SkillDAG success on ALFWorld from 67.10% to 81.43%, improves MemP Proceduralization success from 81.34% to 90.30%, and increases TravelPlanner hard-constraint satisfaction from 64.06 to 76.19. Our code is available at https://anonymous.4open.science/r/learned-relevance-scorer-agent-retrieval-4683/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, skill retrieval, procedural memory, neural reranking, task relevance, agent memory, retrieval-augmented agents, learned relevance scoring

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GL1SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TaskMatch, a lightweight relevance scorer for agent skill and procedural-memory retrieval. It learns from offline LLM-annotated relevance labels using frozen query and candidate embeddings, replacing or augmenting existing retrieval scores. Experiments report improvements in retrieval quality and downstream task performance on SkillsBench, ALFWorld, and TravelPlanner.

Critical strengths
  • S1: The method is lightweight and reuses existing embeddings without additional online LLM calls, making it practical for repeated retrieval during agent execution.
Critical weaknesses
  • W1: The paper offers limited methodological novelty. The claimed move “beyond semantic similarity” largely restates the established motivation for supervised relevance ranking, while the method introduces no distinctive design addressing agent harness-specific retrieval challenges.
  • W2: The experiments insufficiently support the central claim of task-aware relevance beyond semantic similarity. Stronger embedding retrievers and established rerankers are not compared, and execution-related relevance errors are not systematically analyzed.
Questions for authors
  • Q1: How is the reliability of the LLM-generated relevance labels validated, particularly for hard negatives and the required/supportive distinction?
  • Q2: How sensitive is TaskMatch’s performance to the embedding model, its parameter scale, and embedding dimensionality? Have these factors been evaluated under the same retrieval setting?
10 comments
AuthorsOpenReview ↗

Response to Reviewer GL1S: Reliability of LLM-Generated Relevance Labels, Particularly Required/Supportive Distinctions and Hard Negatives

Reproducibility and Cross-Context Stability. Our 33 generic templates × 37 skills yield 1,221 judgments, generated in one run with 100% parsing success; grade 0/1/2 counts are 812/325/84. Templates exclude specific task entities, expert plans, and evaluation data.

GroupingGroupsModal Agreement
Template × skill (≥10 pairs)764100.0%
Identical phase_query × skill (≥8 pairs)1,02697.3%

For find desklamp, alfworld-device-operator receives grade 1 in 55/55 occurrences, while alfworld-goal-interpreter receives grade 0 in 55/55.

Inter-Annotator Agreement. A second model relabeled all 1,221 judgments using the same prompt:

MetricnAgreementCohen's κ
Three-level (0/1/2)1,22178.71%0.578
Required vs. Supportive (reference ∈ {1,2})40970.42%0.400
Required vs. Conflict (reference ∈ {0,2})89682.70%0.446
Reference Label→ 0→ 1→ 2
Grade 2 (Required, 84)0 (0.0%)16 (19.0%)68 (81.0%)
Grade 1 (Supportive, 325)76 (23.4%)220 (67.7%)29 (8.9%)
Grade 0 (Irrelevant/Conflicting, 812)673 (82.9%)138 (17.0%)1 (0.1%)

Thus, none of the 84 reference grade-2 labels is assigned grade 0, and the precision of predicted grade 2 is 68/69 = 98.6%.

Functional Validity: Each Grade Carries Downstream Information. Although absolute label correctness is difficult to measure directly, we test whether each distinction is useful by removing it and measuring performance loss.

Required vs. Supportive. Binarizing grades 1 and 2 causes a substantial drop:

SupervisionMemory Ret@1Skill Ret@1
Three-level (0/1/2)91.0471.66
Merge grades 1+248.75 (−42.29)68.86 (−2.80)
Remove pairwise loss56.63 (−34.41)62.67 (−8.99)

If the grade-1/2 distinction were mainly noise, merging them should dilute noise, not cause a 42.29-point collapse. This indicates non-redundant information about execution order: what is needed now versus what can be done later.

Hard Negatives. Under the same negative-sampling budget:

Negative ConstructionSkill Ret@1Memory Ret@1
Hard (semantically similar, execution-incompatible)71.6691.04
Random59.08 (−12.58)87.46 (−3.58)

If hard-negative grade-0 labels were random noise, they should not systematically outperform random negatives. Their incompatibility is largely objective: heat/cool are mutually exclusive state changes; single-/dual-object placement differs in cardinality; locate/verify differ in execution order; and for find tissuebox, receptacle-finder and object-retriever are lexically similar but trigger different actions. Appendix E.1/E.2 provides the original rankings for independent verification.

Scope of Claims and Mitigations. We do not claim that LLM labels equal expert ground truth or that individual labels are noise-free (the κ values above quantify disagreement). We claim that labels are (i) reproducible under fixed templates, (ii) stable across contexts, (iii) reliable in polarity (zero grade-2→grade-0 errors), and (iv) each level contributes measurable, non-redundant downstream information.

To address grade-0/1 boundary noise, the revision will: (1) apply stricter grade-1 criteria, requiring an explicit prerequisite/supportive or recovery role; otherwise, assign grade 0; and (2) clarify in the main text and Limitations that our method relies on group-level consistency and informativeness, not perfect individual-label accuracy.

AuthorsOpenReview ↗

Response to Reviewer GL1S: Sensitivity to Embedding Models, Model Capacity, and Embedding Dimensions

Response to Reviewer GL1S: Sensitivity to Embedding Models, Model Capacity, and Embedding Dimensions

We have completed all three sensitivity studies under the same retrieval setting (identical frozen embeddings, graded relevance labels, and three random seeds).

(1) Model Capacity: Insensitive to Parameter Count — Model Capacity Is Not the Bottleneck

ConfigurationParametersRet@1MRR
tiny148K70.59 ± 7.3581.48
Paper configuration298K73.53 ± 2.5583.14
wide_1m1.22M75.98 ± 0.8583.92
wide_5m3.52M75.98 ± 3.4084.51
wide_20m7.77M73.53 ± 3.8982.97

As the parameter count varies from 148K to 7.77M (a 52× range), Ret@1 fluctuates only between 70.59 and 75.98, with no monotonic trend (it even drops back to 73.53 at 7.77M).

(2) Embedding Dimensions: TaskMatch Is More Robust Than Fixed Cosine, with Larger Advantages at Lower Dimensions

We applied a shared Gaussian random projection to both task and candidate embeddings (without using PCA), followed by L2 re-normalization, and recomputed the cosine baseline:

DimensionCosineTaskMatchTaskMatch − Cosine
12838.2465.69+27.45
25642.6571.08+28.43
51250.0073.04+23.04
102448.5367.16+18.63
204852.9471.08+18.14
307252.9474.02+21.08

Two conclusions follow:

  1. TaskMatch is more tolerant of dimensionality reduction than cosine: from 128 to 3072 dimensions, cosine improves by +14.70 pp, whereas TaskMatch improves by only +8.33 pp (65.69 → 74.02).
  2. TaskMatch has a larger relative advantage at lower dimensions: +27.45 pp at 128 dimensions versus +21.08 pp at 3072 dimensions. This has practical significance: when deployment systems use smaller embeddings to save storage and bandwidth, TaskMatch offers even greater relative gains with lower integration costs.

(3) Embedding Models

We independently regenerated all task and skill representations using both embedding models (with exactly identical input texts: task_desc and skill documents), ensuring that the comparison is not affected by textual differences:

Methodtext-embedding-3-large (3072d)text-embedding-3-small (1536d)Δ
Cosine51.4747.06−4.41
TaskMatch71.0867.16−3.92
TaskMatch gain over cosine+19.61+20.10+0.49

When switching to a smaller and cheaper embedding model, TaskMatch's absolute performance drops by 3.92 pp, but its gain over the host cosine baseline remains almost entirely unchanged (+19.61 → +20.10, a difference of only 0.49 pp).

In other words, TaskMatch's benefits are decoupled from the capacity of the embedding model: better embeddings improve all methods, while weaker embeddings reduce the performance of all methods, but the learned scorer's advantage over fixed cosine remains consistently present.

Therefore, TaskMatch's effectiveness does not depend on a specific embedding model, embedding dimension, or model-capacity configuration. None of these three variables changes the central conclusion that learned scoring outperforms fixed cosine similarity.

Thank you for answering my questions. My main concerns about methodological novelty (W1) and the evidence supporting the central claim (W2) remain unresolved, so I will keep my original score.

Reviewer_pmkaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Agent systems that look up stored skills or past memories usually rank them by embedding similarity to the query. The paper argues that similarity is not necessarily a useful proxy for the task. The proposed method TaskMatch learns a rerank model with offline examples labeled as required, supportive or irrelevant. Then during online retrieval, TaskMatch adds an additional step with no LLM call leveraging frozen query, skill and memory representations already available to rerank before sending the list to downstream agent. The result shows ALFWorld success with SKILLDAG goes from 67.10% to 81.43%, MEMP Proceduralization from 81.34% to 90.30%, and TravelPlanner hard-constraint score (#HC) from 64.06 to 76.19.

Critical strengths

TaskMatch has a clear and practical design. It changes candidate scoring while retaining the host's frozen embeddings, query generation, and downstream agent. The rerank model is small and requires no additional online LLM calls. So it is easy to adopt and the comparison with cosine at the same step is clean.

TaskMatch includes informative component controls and a careful distinction between retrieval quality and execution success. It evaluates GMF, MLP, and full-model variants, along with graded-label, pairwise-loss, and negative-sampling ablations, using the same data, frozen embeddings, and evaluation protocol. These controls help identify which components contribute to ranking improvements.

Critical weaknesses

The comparisons do not sufficiently isolate TaskMatch's benefit.

  • Table 5 provides useful branch ablations, but no competitive lightweight reranker trained on the same labels and frozen representations. Such a comparison would clarify the advantage of the TaskMatch recipe beyond replacing fixed similarity with learned scoring.
  • The paper retains SkillDAG as a published reference while evaluating TaskMatch locally. The 67.10% → 81.43% gain therefore lacks a same-stack, unmodified SkillDAG control. Downstream results also lack uncertainty estimates. A matched local baseline and repeated or paired evaluations would strengthen attribution.

Algorithmic novelty is limited: Sections 3.2-3.3 adapt NeuMF's GMF/MLP architecture with graded labels and familiar ranking objectives, while Section 2.3 acknowledges existing learned relevance and agent-reranking approaches. The contribution therefore appears primarily to be a lightweight integration into skill and memory retrieval. Although practically useful, the paper does not clearly establish a distinctive methodological advance or empirical insight beyond that integration. Standard components can support a valuable empirical paper, but the additional knowledge contributed here needs clearer substantiation.

Questions for authors

Beyond the branch ablations in Table 5, how does TaskMatch compare with a competitive lightweight supervised reranker trained on the same labels and frozen representations? Please report both retrieval/downstream quality and scoring cost to clarify the benefit of the particular TaskMatch recipe.

Could you report a locally rerun, unmodified SkillDAG baseline using the same agent configuration, prompts, graph behavior, query handling, and step budget as TaskMatch? Repeated or paired evaluations with confidence intervals would help substantiate the headline skill gain and assess the smaller downstream improvements.

What is the main methodological or empirical finding beyond adapting NeuMF and graded supervised ranking to these host systems? The practical integration and matched memory gains are useful; please clarify what additional insight distinguishes TaskMatch from existing learned relevance and agent-reranking approaches.

4 comments
AuthorsOpenReview ↗

Response to Reviewer pmka: Controlled Comparison with Lightweight Supervised Rerankers

We thank the reviewer for this valuable suggestion. We fully agree that controlled comparisons with lightweight supervised rerankers under identical conditions are essential for evaluating TaskMatch's effectiveness.

(1) Retrieval Metrics

Unified Setting: All methods share the same frozen 3,072-dimensional embeddings, graded relevance labels (0/1/2), train/dev/test splits, 140-task end-to-end evaluation, and evaluation codebase. Three random seeds are used for retrieval-side experiments.

MethodModel FamilyRet@1MRRRet@5NDCG@10
Task–skill cosineZero-shot52.9468.0869.8556.57
Linear scorerLinear45.10 ± 0.8567.0988.2466.66
LambdaMARTGBDT69.12 ± 1.4779.9987.3477.34
Low-rank bilinear (r=16)Bilinear73.04 ± 2.2582.7792.5781.64
GMF-onlyNeural72.06 ± 1.4781.6891.5880.72
MLP-onlyNeural74.02 ± 4.2584.0989.1380.85
TaskMatch (Ours)Neural75.00 ± 1.4783.7190.3681.33

Conclusion: TaskMatch is not the best on every metric, but achieves the best Ret@1, with its advantage further reflected in downstream performance (see below).

(2) Downstream Task Performance

Under the same harness, the same 140 tasks, the same LLM, and the same 30-step budget:

MethodSuccess Rate
SkillDAG (without neural reranking)67.86%
SkillDAG + GMF-only72.14%
SkillDAG + MLP-only75.71%
SkillDAG + LambdaMART77.14%
SkillDAG + R3-Reranker-0.6B (zero-shot)70.29%
SkillDAG + TaskMatch81.43%

(3) Computational Overhead of the Scoring Stage

Single-query scoring latency was measured on CPU using batched forward passes over all candidates. Each result is the median of 50 runs.

Method37-candidate pool300-candidate poolPer task (2.13 retrieval calls)
Task–skill cosine0.042 ms0.040 ms0.090 ms
Low-rank bilinear0.085 ms0.250 ms0.181 ms
GMF-only0.090 ms0.238 ms0.192 ms
Linear0.109 ms0.130 ms0.232 ms
MLP-only0.132 ms0.334 ms0.282 ms
TaskMatch0.199 ms0.501 ms0.424 ms
LambdaMART0.435 ms1.993 ms0.926 ms

Key Observation: TaskMatch is 2.18× (37-candidate pool) to 3.98× (300-candidate pool) faster than LambdaMART, with the gap increasing as the candidate pool grows. LambdaMART's computational cost grows faster with the number of candidates: it must construct 8-dimensional handcrafted features for each candidate (cosine similarity, norms, and statistics of element-wise products) and traverse all 500 trees with 15,500 leaves. In contrast, TaskMatch only requires a matrix multiplication followed by a small MLP forward pass. In terms of model size, TaskMatch (1.19 MB, 297K parameters) is also 2.2× smaller than LambdaMART (2.59 MB: 1.80 MB for trees + 0.79 MB for random projections).

Performance of two 0.6B-scale cross-encoder rerankers on the same data:

MethodParametersRet@1Per-pair scoring latency
TaskMatch0.30M75.000.0054 ms (0.199 / 37)
SkillRouter-Reranker-0.6B600M30.8822.00 ms
R3-Reranker-0.6B (zero-shot)600M29.4184.62 ms

In other words, TaskMatch uses 2,000× fewer parameters, achieves 4,000–15,000× faster scoring, and obtains approximately 2.5× higher Ret@1.

(4) TaskMatch's Distinctive Advantages: Our Positioning

Taken together, the three tables support the following combination of advantages, rather than a claim that TaskMatch is the best on every metric:

  1. Highest end-to-end success rate (81.43%), with a statistically significant improvement over the host retriever (+13.57 pp, p = 0.0066).
  2. End-to-end differentiability: TaskMatch is a neural network that can be jointly optimized with upstream representations, whereas LambdaMART is a tree ensemble and is non-differentiable, preventing gradient propagation.
  3. No feature engineering: LambdaMART relies on 8-dimensional handcrafted features and a fixed 64-dimensional random projection, requiring feature redesign when switching datasets. TaskMatch only consumes the host system's existing E(q)E(q) and E(c)E(c).
  4. Compared with high-capacity cross-encoders: TaskMatch achieves 3–4 orders of magnitude lower latency with higher retrieval accuracy, making it practical to invoke repeatedly at every retrieval event during long-horizon agent execution.
AuthorsOpenReview ↗

Response to Reviewer pmka: Paired End-to-End Evaluation and Statistical Uncertainty

We fully acknowledge the reviewer's criticism and must first recognize an important fact: the SkillDAG row in our paper was indeed labeled as a published reference. The original sentence in the revised manuscript reads:

"...with matched local evaluations of the scorer variants, while the SkillDAG row is retained as the published reference."

Therefore, the reviewer interpreted this correctly: that row was not based on a local paired evaluation at the time.

(1) We Have Conducted a Local Paired Reproduction, with Results Almost Identical to Those Reported in the Paper

We reran the original SkillDAG under the same harness, the same list of 140 tasks, the same LLM, and the same execution budget:

MethodSuccess Rate
Original SkillDAG (local rerun)67.86%
SkillDAG + TaskMatch (local)81.43%
Originally cited value in the paper67.10%

The locally rerun result of 67.86% differs from the originally cited value of 67.10% by only 0.76 percentage points, supporting the reliability of the originally reported result as well as the validity of our reproduction.

(2) Statistical Uncertainty Estimation

We conducted task-level paired statistical tests on the 140 matched tasks:

  • Paired bootstrap (10,000 resamples): Δ = +13.57 pp, with a 95% CI of [+4.29, +22.86].
  • Exact McNemar's test: p = 0.0066.

Thus, under the paired evaluation design, the improvement is statistically significant at the α = 0.01 level.

(3) Revision Commitments

  1. We will replace the published-reference value of 67.10% with the locally measured value of 67.86% in the main text and explicitly identify it as a paired local rerun.
  2. We will add Wilson 95% confidence intervals to the ALFWorld results table.
AuthorsOpenReview ↗

Response to Reviewer pmka: Core Methodological Findings and Distinctive Insights over Prior Work

1. Frozen-embedding interaction can outperform cross-encoders without re-encoding. Same frozen embeddings, labels, splits, and 68-task test set:

MethodParamsRet@1Latency/pair
TaskMatch0.30M75.000.0054 ms
SkillRouter-Reranker-0.6B600M30.8822.00 ms
R3-Reranker-0.6B (zero-shot)600M29.4184.62 ms

TaskMatch uses 2,000× fewer parameters, is 4,000–15,000× faster, and has ~2.5× higher Ret@1. Agent queries (e.g., find desklamp, put a fork in countertop) and skills describe capabilities. Matching tests capability-operation compatibility, not token alignment; cross-encoder interactions are unnecessary and may overfit wording. The bottleneck is relevance definition, not representation capacity: host embeddings already exist; learning the scorer suffices.

2. MRR/Recall@K alone are insufficient for agent model selection.

MethodMRR↑Recall@5↑Ret@1↑End-to-end SR↑
MLP-only84.0989.1374.0275.71%
Low-rank bilinear82.7792.5773.04—
TaskMatch83.7190.3675.0081.43%

MLP-only has highest MRR but lowest SR; bilinear has highest Recall@5 without downstream gains. Rank 1→2 for the first relevant item costs only 0.5 MRR; Recall@K ignores order. Yet agents consume Top-1/Top-2 identities: retrieving cool instead of required heat can cause failure. Thus evaluate Top-1 hit rate + Top-K harmful-item rate; TaskMatch Bad@3 = 25.49 (lowest).

3. The intermediate “supportive” grade is essential. Unlike binary IR, agent retrieval distinguishes required skills from assistance, prerequisites, verification, or recovery.

SupervisionSkill Ret@1Memory Ret@1
Three-level (0/1/2)71.6691.04
Binary (0 vs. 1+2)68.8648.75 (−42.29)
No pairwise loss62.6756.63 (−34.41)

Binarization drops Memory Ret@1 by 42.29 points: distinguishing “necessary now” from “useful later” preserves execution-order information, unlike binary classical IR.

4. Execution-incompatible hard negatives outperform random negatives.

NegativesSkill Ret@1Memory Ret@1
Hard (Ours)71.6691.04
Random59.08 (−12.58)87.46 (−3.58)

Agent skills contain textually similar but execution-incompatible pairs (heat/cool, locate/verify, single-/two-object placement). These are agent-specific structural conflicts requiring explicit hard negatives.

5. Task-aware relevance is independent of the host retriever. The same fθ(E(q),E(c))f_\theta(E(q),E(c)) and training pipeline improve SkillDAG (graph retrieval with dependency/prerequisite/conflict edges) and MemP (cosine memory retrieval): ALFWorld Ret@1 57.90→86.03; Memory Ret@1 73.84→91.04. This supports plug-and-play learning independent of the retrieval operator.

Prior-work positioning

  • NCF/NeuMF (He et al., 2017): user/item IDs; TaskMatch uses frozen content embeddings and scores unseen skills (59.31 vs. 34.80 for ID methods).
  • SkillRouter/R3-Reranker: 0.6B raw-text cross-encoders; TaskMatch avoids re-encoding (0.3M; 3–4 orders lower latency).
  • LambdaMART/GBDT: handcrafted cosine/norm/element-wise-product features + random projection; TaskMatch needs no feature engineering and is 2.2–4.0× faster.
  • ColBERT/DRMM/MatchPyramid: token-level interaction; TaskMatch uses no raw text; embeddings are precomputable/cacheable.
  • Original MemP: fixed cosine; TaskMatch changes scoring only, not memory construction, storage, or updating.

Controlled MemP evidence. MemP uses cosine retrieval. Only the scorer changes; the GPT-4o agent, prompts, action budget, memory format, and execution code are unchanged.

MemoryQuery cosine+ TaskMatchΔ
Proceduralization (ALFWorld SR)81.34%90.30%+8.96 pp (12 tasks)
Proceduralization (TravelPlanner #HC)64.0676.19+12.13

We retain Script’s negative result: gains depend on direct usability after retrieval. Proceduralization stores executable procedures; Script stores trajectory summaries requiring agent inference. This explains when gains occur.

Summary. Agent retrieval needs neither large re-encoders, ID memories, nor token interactions, but a 0.3M scorer trained with graded execution-relevance labels on frozen embeddings. Three principles: (i) preserve the supportive grade, (ii) construct execution-incompatible hard negatives, (iii) select by end-to-end SR, not MRR. Ablations support all (−42.29 pp; −12.58 pp; MRR/end-to-end ranking reversal).

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_2jEJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies retrieval in LLM agent harnesses, arguing that semantic similarity between a query and a skill or procedural memory does not necessarily reflect its usefulness for task execution. The authors propose TaskMatch, a lightweight learned relevance scorer that reranks candidates using frozen query and candidate embeddings. The model combines Generalized Matrix Factorization (GMF) and MLP interaction branches, trained with pointwise and pairwise objectives on offline LLM-generated relevance labels. TaskMatch is integrated into two existing retrieval systems, SkillDAG and MemP. Experiments across SkillsBench, ALFWorld, and TravelPlanner show improvements in both retrieval metrics and task performance.

Critical strengths
  1. A practical and well-motivated problem. The paper identifies a relevant limitation of similarity-based retrieval in agent systems: semantically related skills or memories are not necessarily appropriate for the operation being performed. Learning relevance from task requirements is a sensible direction.

  2. Lightweight and minimally invasive integration. TaskMatch operates directly on frozen embeddings and modifies only candidate scoring. Its small parameter count (0.15–0.30M) and low measured inference overhead make it practical for repeated retrieval in agent workflows.

  3. Mostly encouraging retrieval and downstream results. The paper reports retrieval improvements and meaningful task-success gains across multiple settings.

Critical weaknesses
  1. Limited methodological novelty and insufficient comparisons with learned rerankers. The core architecture is adapted from the established NeuMF formulation, while graded relevance supervision and pairwise ranking are standard learning-to-rank techniques. The main comparisons are against existing similarity-based retrieval mechanisms rather than competitive learned task-aware rerankers under matched supervision. Although the GMF/MLP ablations are useful, stronger comparisons are needed to establish whether the proposed architecture offers advantages beyond applying a conventional supervised reranker to agent retrieval. Also the experiment lack sufficient baselines and only include 2-3 very basic ones.

  2. The supervision does not directly capture execution utility and can use information unavailable to the scorer. TaskMatch relies on LLM-generated relevance labels rather than observed contributions to task success. In the skill setting, the offline judge receives the full task workflow in addition to the operation-specific query, while TaskMatch receives only the query and candidate embeddings at inference time. Thus, relevance judgments may depend on task context unavailable to the trained scorer. The paper does not sufficiently examine label reliability or how this mismatch affects retrieval on ambiguous or context-dependent queries. It's unclear why we should use LLM generated labels.

  3. The strongest skill-execution comparisons lack paired evaluation and statistical validation. The paper states that the skill-retrieval baseline results are taken from the original papers, whereas TaskMatch results come from local runs. This limits confidence that the end-to-end improvements arise solely from the changed scoring function under otherwise identical execution conditions.

  4. Generalization beyond the supervised repositories remains unclear. The relevance supervision is constructed separately for each target setting, and the skill and memory scorers are trained independently. Although the paper establishes improvements within these settings for limited baselines, it does not demonstrate transfer to unseen skill libraries, new memory collections, or substantially different task distributions without new annotation.

Questions for authors
  1. How does TaskMatch compare with stronger supervised reranking baselines under matched training conditions? Since TaskMatch primarily adapts an established NeuMF architecture, could the authors compare against conventional pairwise rankers, bilinear query–candidate scoring, or lightweight neural rerankers trained on the same frozen embeddings and relevance labels? In particular, baselines worth considering includes but not limited to LambdaMART (LightGBM Ranker), Low-rank bilinear scorer, R3-reranker, SkillReranker.

  2. How reliable are the LLM-generated relevance labels, and does the additional context available to the judge introduce a supervision mismatch? The skill-labeling prompt provides the full workflow to the judge, while TaskMatch operates on the local retrieval query alone. Can the authors measure label consistency when identical or similar local queries appear in different task contexts? An ablation where labels are generated using only the information available to TaskMatch at inference time would help determine how much the richer teacher context contributes.

  3. Can the authors provide paired end-to-end evaluations and statistical uncertainty estimates? The procedural-memory comparisons are locally rerun under the same MemP pipeline, which is useful. Could the authors provide similarly controlled reruns of SkillDAG with and without TaskMatch using identical agent configurations, evaluation tasks, and execution budgets?

  4. How well does TaskMatch generalize to new skills, memories, and task distributions? Does the learned scorer transfer to a different skill repository or an expanded memory bank without retraining? Could the authors evaluate retrieval when some candidate skills or task families are excluded from supervision and introduced only at test time?

  5. What explains the gap between retrieval improvements and downstream agent success? Table 5 shows that the MLP-only variant achieves higher Ret@1 and MRR than the full TaskMatch model, yet has lower task success (75.71% versus 81.43%). Similarly, Script memory gains substantially in intrinsic retrieval quality but improves ALFWorld success by only 0.74 points.

11 comments
AuthorsOpenReview ↗

Response to Reviewer 2jEJ – Question 1: Supervised Reranking Baselines

We thank the reviewer for suggesting stronger supervised reranking baselines. We agree that controlled comparisons using identical embeddings, labels, and splits are needed to establish TaskMatch's benefits beyond cosine retrieval, and have conducted additional experiments.

1. Evaluation protocol. All lightweight scorers use the same frozen text-embedding-3-large vectors (3,072-D), graded labels (required/supportive/irrelevant), pairwise preferences, train/dev/test splits, and seeds (2026/2027/2028). Training and evaluation share one codebase. Each family uses appropriate hyperparameters; LambdaMART uses LightGBM lambdarank (500 trees, 31 leaves, learning rate 0.05).

2. Intrinsic retrieval (ALFWorld-37; 68 test tasks; 3 seeds). All metrics are percentages; Ret@1 shows mean ± SD.

MethodFamilyRet@1MRRRet@5NDCG@10
Task–skill cosineZero-shot52.9468.0869.8556.57
Linear scorerLinear45.10 ± 0.8567.0988.2466.66
LambdaMARTGBDT69.12 ± 1.4779.9987.3477.34
Low-rank bilinear (r=16r=16)Bilinear73.04 ± 2.2582.7792.5781.64
GMF-onlyNeural72.06 ± 1.4781.6891.5880.72
MLP-onlyNeural74.02 ± 4.2584.0989.1380.85
TaskMatchNeural75.00 ± 1.4783.7190.3681.33

TaskMatch achieves the best Ret@1 (+5.88 pp over LambdaMART), the top-ranked candidate being especially relevant to agent retrieval. MLP-only achieves the best MRR, while bilinear scores best on Ret@5/NDCG@10. The linear model underperforms cosine (45.10 vs. 52.94), highlighting the value of query–skill interactions. These baselines span linear, bilinear, GBDT, shallow neural, and dual-branch models.

3. End-to-end evaluation (same 140 ALFWorld tasks).

MethodIntrinsic Ret@1Success rate
SkillDAG (baseline)57.9067.86%
+ LambdaMART69.1277.14%
+ TaskMatch75.0081.43%

LambdaMART improves SkillDAG by 9.28 pp, confirming the value of supervised ranking, but trails TaskMatch by 4.29 pp. The intrinsic and downstream rankings are consistent.

4. Is the gain merely due to the training objective? We implemented R3-inspired graded ListNet on the same 297K-parameter architecture, frozen embeddings, and labels:

ObjectiveRet@1MRR
Pointwise + pairwise (TaskMatch)73.5383.14
ListNet (R3 objective)51.9668.05
ListNet only60.7873.62
Pointwise only70.5981.14

The R3-inspired listwise variant reduces Ret@1 by 21.6 pp, with large seed variation (39.71–61.76). Thus, the interaction design and objective work together; simply adopting a listwise loss does not explain the gain.

5. R3-Reranker and SkillReranker. We obtained the official tencent/R3-rerank-0.6b (arXiv:2606.03565) and pipizhao/SkillRouter-Reranker-0.6B (arXiv:2603.22455) checkpoints. Both are cross-encoders operating on query–skill text, not our cached frozen sentence embeddings. They therefore cannot satisfy the identical-embedding constraint, although cross-architecture comparisons remain informative.

Without target-domain fine-tuning, SkillRouter-Reranker scores 30.88 Ret@1 on ALFWorld-37 (vs. cosine's 52.94), plausibly reflecting transfer from code-skill libraries to household skills. R3-Reranker obtains 70.29% end-to-end success (vs. TaskMatch's 81.43%). Reported inference measurements are 22 ms for R3 and 0.199 ms for TaskMatch, corresponding to an approximately 4,000-fold difference under our reported evaluation setting; the per-candidate/per-query timing basis must be distinguished before claiming a speedup factor. Cross-encoders require a separate text-processing pipeline, whereas TaskMatch reuses cached embeddings without extra online LLM calls.

SkillReranker (arXiv:2607.06283) is distinct from SkillRouter-Reranker. We could not find an official complete implementation. Although its Qwen3-Reranker-0.6B backbone is public, faithful reproduction would require task decomposition, execution-graph construction, and adaptive selection. These extra structural inputs and selection mechanisms differ from our controlled frozen-embedding reranking setting. We plan to discuss this distinction in Related Work and prioritize reproducible supervised baselines, including RankNet-style pairwise neural scoring.

6. Revision plan. Under matched frozen representations, labels, splits, and seeds, TaskMatch has the highest Ret@1 and exceeds LambdaMART by 4.29 pp in end-to-end success. We will add the scorer-family and LambdaMART end-to-end tables, with protocols, hyperparameters, and reproducibility details in the appendix.

AuthorsOpenReview ↗

Response to Reviewer 2jEJ – Question 2: Label Reliability and Context Mismatch

We thank the reviewer for this concern. Our manuscript insufficiently clarified TaskMatch's inference-time inputs, potentially suggesting that the labeler sees the full task while the scorer sees only a local query. This is a presentation issue. We provide empirical evidence below, including the requested context ablation.

1. Labeling Inputs: Task-Agnostic Capability Templates

Labels are not assigned from instance-specific full tasks. The corpus uses 33 (task type × phase) templates, each containing five fields: task_type (e.g., look_at_obj_in_light), generic_full_workflow (e.g., "locate a target object and a light device, then examine the object under the activated light"), phase_name, generic_phase_objective, and phase_group.

The data files verify that three information types are excluded: uses_specific_task_entity=False, uses_expert_plan=False, and uses_eval_data=False. Thus, the labeler sees abstract capability requirements, not specific task entities, expert action steps, or evaluation-set content.

2. Label Stability Across Contexts

If labels depended on full-task semantics, the same (phase, skill) pair should vary substantially across task instances. Instead:

  • Template × skill: 1,018 groups (764 with ≥10 pairs); modal agreement is 100.0% (mean and median).
  • Exact phase_query × skill: 1,026 groups (≥8 pairs); modal agreement is 97.3%.

For find desklamp, alfworld-device-operator is grade 1 in 55/55 occurrences; alfworld-goal-interpreter is grade 0 in 55/55; and alfworld-locate-target-object is grade 1 in 55/55. This suggests labels primarily reflect local phase requirements × skill capabilities, with task context serving mainly to disambiguate.

3. Requested Context Ablation

We varied only the visible context, keeping MiniMax-M2.7 and all other prompt components fixed.

A. Phase Query Only (Task Type Also Removed). Across 196 phases and 2,397 comparable pairs:

MetricValue
Exact agreement63.2%
Quadratic weighted κ0.307
Positive/negative agreement66.75%
Positive-label rate43.89% → 16.15%
Mean absolute grade shift0.50

A short phrase such as "find desklamp" is insufficient to determine phase-specific skill relevance. Thus, task-level context is necessary for annotation, as the reviewer suggests.

B. Task Type + Phase Objective (Verbatim Workflow Removed). Across 30 templates and 882 (template, skill) judgments:

MetricValue
Exact agreement82.65%
Quadratic weighted κ0.791
Severe reversals (grade 0 ↔ 2)0.57% (5/882)
Adjacent-grade reversals (0↔1, 1↔2)16.78%
Positive-label rate35.49% → 41.16%

The key signal is task type + phase objective, not the verbatim workflow: after removing the workflow, κ remains 0.791, with only 0.57% severe reversals. The remaining 16.8% are adjacent-grade differences; grade 1 ("supportive assistance/prerequisites/recovery") naturally has ambiguous boundaries with grades 0 and 2. The positive-label rate also remains similar (35.5% → 41.2%), indicating that the discriminative signal is largely preserved.

4. Task-Level Context Is Available at Inference

When enabled, skilldag/graph.py::_ncf_context_query() constructs the retrieval input as:

Full task:\n{full_task}\n\nCurrent phase/query:\n{query}

Thus, it provides the full task description + current phase, structurally aligned with the labeling prompt. Under this input view, the alleged information mismatch does not exist at the structural level.

5. Effectiveness Without the Full-Task View

Even under the reported configuration without the full-task view, on the same 140 ALFWorld tasks, LLM, and candidate pool:

MethodSuccess Rate
SkillDAG67.10%
SkillDAG + TaskMatch81.43% (+14.33 pp)

Such a gain should not be observed if the annotation–inference mismatch were severe enough to invalidate the supervision signal.

6. Summary and Revision

  1. Labels use task-agnostic capability templates, excluding specific entities, expert plans, and evaluation data.
  2. Labels are stable across contexts (100.0% / 97.3%) and primarily reflect (phase, skill) capability matching.
  3. We will clarify TaskMatch's inference-time inputs in the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer 2jEJ – Question 3: Paired End-to-End Evaluation and Statistical Uncertainty

We fully acknowledge the reviewer's criticism, as well as one fact that must be explicitly recognized: the SkillDAG row in our paper was indeed labeled as a published reference. The original sentence in the revised manuscript reads:

"...with matched local evaluations of the scorer variants, while the SkillDAG row is retained as the published reference."

Therefore, the reviewer is entirely correct: that row was not based on a locally paired evaluation at the time.

1. We Have Conducted a Local Paired Reproduction, with Results Almost Identical to Those Reported in the Paper

We reran the original SkillDAG under the same harness, the same list of 140 tasks, the same LLM, and the same execution budget:

MethodSuccess Rate
Original SkillDAG (locally rerun)67.86%
SkillDAG + TaskMatch (local)81.43%
Previously cited value in the paper67.10%

The locally rerun result of 67.86% differs from the previously cited value of 67.10% by only 0.76 percentage points, supporting the reliability of the previously cited result as well as the validity of our reproduction.

2. Statistical Uncertainty Estimation

We conducted task-level paired statistical tests on the 140 matched tasks:

  • Paired bootstrap (10,000 resamples): Δ = +13.57 pp, with a 95% CI of [+4.29, +22.86].
  • Exact McNemar's test: p = 0.0066.

Therefore, under the paired evaluation design, the improvement is statistically significant at the α = 0.01 level.

3. Revision Commitments

  1. We will replace the published-reference value of 67.10% with the locally measured value of 67.86% in the main text and explicitly identify it as a paired local rerun.
  2. We will add Wilson 95% confidence intervals to the ALFWorld results table.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.