acceptodds
Under review as a conference paper at ICLR 2027

SkillLineage: Black-Box Provenance Auditing of Agent Skills Across Language Models

Abstract

Agent skills package procedures, scripts, and reference materials for reuse across language models, but this portability makes their origin difficult to audit once they are copied, rewritten, or reconstructed behind another agent’s API. We study black-box skill provenance auditing: determining whether an opaque deployment derives from a protected skill or its outputs, rather than from an independently developed solution to the same task. The challenge is that provenance is easily confounded by two effects: independent same-task skills can exhibit similar functionality, while the same skill can appear different when deployed on another model. We introduce SKILLLINEAGE, to our knowledge the first passive fingerprinting framework for black-box agent skill provenance auditing, based on local behavioral correspondence. Using an owner-controlled model, it selects skill-sensitive contexts and queries the suspect with matching response prefixes. The resulting profile records continuation agreement at selected response positions and is compared with references generated with and without the protected skill. The audit uses only returned text, without prior modification of the protected skill or knowing its host model. Across five released skills, 240 deployments on three open-source models, and four black-box APIs, SKILLLINEAGE achieves a 0.9977 AUROC on the three primary skills using only Llama-based references.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent Skills, Black-Box Provenance Auditing, Behavioral Fingerprinting, Large Language Models, Intellectual Property Protection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iw6JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper formulates black-box skill provenance auditing as a hypothesis test:

  • H1H_1: the suspect skill was derived from the protected skill or its outputs.
  • H0H_0: the suspect skill is absent, unrelated, or independently developed for the same task.

The proposed audit has three stages:

  1. Probe selection. The owner writes eight skill-specific task questions. Using a Llama reference model with and without the protected skill, the method selects six response windows per question where the protected skill increases agreement with the owner’s target continuation.

  2. Fingerprint construction. For each selected window, the suspect endpoint is queried under the same owner prefix. Twenty independently restarted next-token observations are collected per window. Each returned fragment is compared with the owner target using all-MiniLM-L6-v2 cosine similarity. The resulting 48 values form a normalized fingerprint.

  3. Provenance scoring. The suspect fingerprint is compared with the protected-skill and no-skill reference fingerprints. The score is the difference between the two cosine similarities, with a positive score interpreted as source-related.

The evaluation uses five SkillsBench packages:

  • PDDL planning,
  • PCAP/network analysis,
  • XLSX spreadsheet operations,
  • 3D mesh analysis,
  • radar vital-sign extraction.

The source-related candidates include the original skill, a paraphrased skill, a Daydreaming reconstruction, and a second skill-stealing reconstruction. The negative candidates include independently generated same-task skills, wrong-task skills, and a no-skill deployment.

For the main open-source evaluation, the reference model is Llama-3.1-8B-Instruct and the host models are Qwen3-8B, InternLM2.5-7B, and GLM-4-9B-0414. The paper reports 240 deployments across these three hosts. It also evaluates DeepSeek-V4.1-Flash, GLM-4.7, GPT-4o-mini, and Gemini-2.5-Flash through API deployments.

The main reported result is a 0.9977 macro pooled-host AUROC on PDDL, PCAP, and XLSX. The supplementary five-skill results are 1.0000 for PDDL, 1.0000 for PCAP, 0.9931 for XLSX, 0.9954 for Mesh, and 0.9167 for Radar. The paper compares against Prompt Detective, Model Equality Testing, LLMmap, ZeroPrint, and BBF, and reports that SkillLineage substantially outperforms those adapted baselines.

Critical strengths

Strength 1: A timely and well-defined problem formulation

The paper isolates a meaningful problem that is not equivalent to model fingerprinting or prompt membership inference. Explicitly including independently developed same-task skills in H0H_0 is a strong choice because it addresses the most important confound: correct task behavior does not necessarily imply shared provenance.

The paper also responsibly states that source-relatedness does not itself imply unauthorized reuse. That distinction is important for any practical provenance auditing system.

Strength 2: A coherent and interpretable audit design

The three-stage design is easy to reason about:

  • select skill-sensitive regions,
  • preserve local correspondence,
  • compare against both protected-skill and no-skill references.

The ablations show that the components are not entirely redundant. The cross-host evaluation and the additional API experiments also suggest that the method captures a real signal on the authors’ benchmark rather than simply identifying the host model.

Critical weaknesses

Weakness 1: The near-perfect results are not demonstrated to generalize

The central results rely on a small and correlated candidate set, fixed owner-selected probes, source-specific response positions, and LLM-generated negative controls. The paper does not include held-out tasks or skills, human-authored independent implementations, adversarial mimics, or a leave-one-skill-out analysis.

The headline 0.9977 AUROC also emphasizes the three “primary” skills, while the five-skill results include Radar at 0.9167 AUROC and an 0.8333 binary accuracy at the fixed threshold. The reported numbers are promising, but they should not be presented as evidence of general black-box provenance auditing without a much stronger generalization study.

Weakness 2: The audit interface and provenance interpretation are too fragile

The method actively forces the suspect system to continue owner-generated prefixes, often one token at a time. This is a strong interface assumption and is not equivalent to observing normal agent behavior. The paper does not test adaptive suspects, response randomization, role wrappers, free-running continuations, tool calls, API updates, or output sanitization.

Moreover, the method measures behavioral similarity, while the paper’s hypothesis concerns historical derivation. Without adversarially strong independent alternatives or an explicit identifiability analysis, the proposed binary decision should be interpreted as a similarity signal rather than a provenance determination.

Questions for authors

Clarifications that could materially affect the current assessment

  1. Probe selection and held-out evaluation: Were the eight probes and six windows for each skill fixed before any candidate outputs were inspected? Can you provide leave-one-task-out, held-out-task, or held-out-procedure results? If not, why should the current numbers be interpreted as generalization rather than source-specific query fitting?

  2. Missing experimental artifacts: Where are the complete probe questions, owner responses, selected window coordinates, source-skill versions, and exact reference traces for PCAP, XLSX, Mesh, and Radar? Appendix C.3 appears to provide only the PDDL questions, which prevents reproducing most of the headline experiments.

  3. Independent negative construction: How exactly were the GLM- and DeepSeek-generated independent skills produced? What prompts, seeds, system instructions, and source information were available? Were they checked for contamination from SkillsBench or the protected skill? Can the authors evaluate expert-authored and style-matched independent skills?

  4. API protocol: For APIs without native assistant prefilling, what exact role and message format was used to request continuation of the owner prefix? How were system prompts, tokenization, whitespace, retries, safety refusals, and output truncation handled? Section 4.2 describes independently restarted next-token requests, while Section 5.5 describes up to 20 generated tokens per query; what is the exact API procedure?

  5. Statistical uncertainty: What are the confidence intervals and clustered bootstrap estimates when clustering by skill artifact, task, and host? How much does the result vary across repeated runs, random seeds, temperature settings, API dates, and model snapshots?

  6. Representation sensitivity: Why is MiniLM sentence embedding cosine similarity applied to individual token or short-fragment outputs? How does SkillLineage compare with exact token matching, BPE-aware lexical similarity, character-level similarity, other encoders, and multi-token continuation representations?

Suggestions for a stronger revision or future work

  1. Evaluate against held-out skills and procedures, expert-written same-task skills, and independent skills explicitly optimized to evade the selected probes.

  2. Add adaptive-suspect experiments in which the endpoint knows the audit questions, detects prefix probing, changes decoding behavior, or post-processes returned text.

  3. Report calibrated precision, recall, false-positive rates, and uncertainty at realistic prevalence levels rather than relying primarily on AUROC and a fixed zero threshold.

  4. Match all baselines by query and token budget, and include simple non-learned baselines such as exact token agreement, lexical overlap, and nearest-neighbor correspondence.

  5. Test free-running responses, tool-use traces, role wrappers, randomized decoding, API model updates, and cases where the skill is internalized into model parameters.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iw6J (1/3): Generalization, Probe Selection, and Artifacts

Thank you for the detailed review. As we understand it, your core worry was that the near-perfect numbers could come from a small and possibly tuned setup: few skills, probes and windows that may have been chosen after seeing outputs, and too little detail for others to check. We agree the paper needed more evidence here. We have since tested six further skills, each under a protocol frozen before its outputs, released the audit inputs, and filled in the missing procedures and uncertainty estimates. We also accept your point that the score shows similarity rather than derivation, and have reworded the claims. Here is our detailed response to each question.

W1: Generalization and the headline number.

Concern: The near-perfect results rest on a small, correlated candidate set with LLM-generated negatives, and there are no held-out skills, human-authored independent skills or adversarial mimics. The headline 0.9977 covers only the three primary skills, while Radar reaches 0.9167.

Response: We agree that the headline should cover all five skills, and we now add six more skills to test generalization. The paper does already include a human-written independent skill (PracticalSwan, Appendix), discussed further in Q3.

  • Headline. We now report the five-skill AUROC (0.981) instead of the three-skill 0.998.
  • Radar. It is our weakest skill (AUROC 0.917, 95% CI [0.696, 1.000]). Its API results (AUROC 0.60–0.88) were missing from Table 3 and are now added.
  • Generalization. We tested six more SkillsBench skills, each drawn once with a fixed seed from a pinned revision. Each has 13 candidates on three hosts (Qwen, InternLM, GLM): Original and a controlled paraphrase as positives; five independent, five wrong-task and No-Skill as negatives. Each row pools the three hosts (6 positive, 33 negative deployments):
SkillAUROCBAcc@0 (%)TPR@FPR=0 (%)TPR@FPR≤5% (%)
transaction-protocol-reasoning1.000100.00100100
video-processor1.00098.48100100
pymatgen1.00090.91100100
python-parallelization1.000100.00100100
rl-post-training1.000100.00100100
jax-skills1.000100.00100100

BAcc uses the fixed zero threshold; TPR uses empirical ROC thresholds.

  • Human-written and adversarial alternatives. Human-written independent skills are covered in Q3. Independent skills optimized to evade our probes are not yet tested; we leave them to future work.

Revision: Section 5.2 and Table 1 now report all five skills, Table 3 adds Mesh and Radar, and Appendix E.2 adds the six skills.

Q1: Were probes and windows fixed before seeing candidate outputs?

Question: Were the eight probes and six windows for each skill fixed before any candidate outputs were inspected? Can you provide leave-one-task-out, held-out-task or held-out-procedure results? If not, why should the numbers be read as generalization rather than source-specific query fitting?

Response: Yes, both were fixed in advance, and the held-out checks below keep AUROC at 0.896 or higher.

  • Windows: yes. Only Llama with/without the protected skill is used.
  • Probes: yes. All eight were fixed before any audit data was collected.
  • Held-out probes. Either four-probe half gives AUROC ≥0.986: PDDL 0.986 / 1.000; PCAP 0.986 / 0.998.
  • Leave-one-probe-out. Dropping any one of the eight probes keeps AUROC at 1.000 (PDDL), 0.991 (PCAP), 0.958 (Mesh), 0.924 (XLSX) and 0.896 (Radar) or higher.
  • Held-out skills. The six skills in W1 were each audited with probes and windows frozen before any of their outputs; all reach AUROC 1.000.

Revision: Appendix E.4 adds the leave-one-probe-out and held-out-probe results.

Q2: Missing artifacts.

Question: Where are the complete probe questions, owner responses, selected window coordinates, source-skill versions and exact reference traces for PCAP, XLSX, Mesh and Radar? Appendix C.3 seems to give only the PDDL questions.

Response: All of these are now public for all eleven skills (the original five and the six new ones): probes, owner responses, window positions, reference traces, and file hashes of each skill version. Link: https://anonymous.4open.science/r/skilllineage-rebuttal-artifact-2236/

Revision: Appendix C.3 now links the released artifact.

AuthorsOpenReview ↗

Response to Reviewer iw6J (2/3): Independent Skills and API Protocol

Q3: How were the independent skills built?

Question: How exactly were the GLM- and DeepSeek-generated independent skills produced (prompts, seeds, system instructions, available source information)? Were they checked for contamination from SkillsBench or the protected skill? Can expert-authored and style-matched independent skills be evaluated?

Response: Two LLMs wrote the independent skills from a short public task description, without seeing the protected skill, and these skills overlap little with it. Two human-written skills are also ranked below every source-related deployment.

  • Generation (original five). GLM-4.7 and DeepSeek-V4-Flash wrote them from a short public task description (e.g., for XLSX: create, edit, recalculate and check workbooks with openpyxl, pandas and LibreOffice). Each skill is one API sample at temperature 1.0 with no fixed seed. The system prompt asks the model to write a self-contained skill from the public capability specification given by the user, and states that no protected skill, probe or suspect output is provided; indeed, the generators never saw any of these.
  • Overlap check. The independent skills share at most one 12-word sequence with the original (none for PDDL). Shared 8-word sequences are generic code, such as looping over worksheet rows. Paraphrased copies, for comparison, share 11–56%. We cannot rule out that the generators saw public SkillsBench skills during pretraining.
  • Human-written skills. A public PDDL skill by other authors (pddl-copilot) is rejected on all three hosts. A public XLSX skill (PracticalSwan) scores just above zero on two hosts but below every source-related XLSX deployment. Both were checked after earlier pilots on them, so we call them retrospective.
  • Functionally matched skills. The independent skills match the protected skill in function, not only in task: on XLSX, Original and five independent skills pass the same four of seven executable tasks with identical pass/fail patterns, yet Original scores higher than every independent skill (Section 5, provenance vs. functionality). Skills that also imitate the protected skill's wording are future work.

Q4: API protocol.

Question: For APIs without native assistant prefilling, what exact role and message format requests the continuation? How were system prompts, tokenization, whitespace, retries, safety refusals and output truncation handled? Given that Sections 4.2 and 5.5 differ, what is the exact API procedure?

Response: We agree the paper left out these details, and the revision now gives the full procedure. In short, APIs without prefill receive the owner's partial answer through ordinary chat messages, and every output is scored as returned, never filtered or retried based on its content.

  • Message format. Every request starts the same way: the suspect's own deployment context as the system message, then the probe task as a user message. The owner's partial answer is then attached in one of three ways:
    • DeepSeek: as an assistant message with DeepSeek's native prefix flag, so the model continues it directly.
    • GLM-4.7: as an assistant message, followed by a fixed user message: "Continue the immediately preceding assistant answer from exactly where it stops. Do not restart, summarize, repeat, or add a preamble. Output only the next continuation."
    • GPT-4o-mini and Gemini: inside a second user message: "Continue the incomplete assistant answer below for the task above. Return only the next 20 tokens to append: do not repeat, restart, or add a preamble. Your response is concatenated directly after its final character, even if a word is cut in half. Answer so far: <owner's text>"
  • Tokenization, whitespace, retries, refusals and truncation.
    • Retries: We resend a request only when it fails for a technical reason: a network error, an answer from a different model than the one requested, or (for XLSX on GLM) a badly formatted answer. We never resend because of what the answer says.
    • Refusals and empty outputs: We keep them and score them as they are. They can only make the suspect look less similar to the owner.
    • Tokenization and whitespace: We compare returned text rather than tokens, so differences between tokenizers do not matter. We only remove leading and trailing spaces.
    • Truncation: Each answer is limited to 20 tokens and is not shortened further.
  • Exact procedure. We send one request for each of the 48 windows, using deterministic decoding (temperature 0) and at most 20 tokens. We compare the returned text, as a whole, with the owner's text for that window. The Llama references are collected in the same way.

Revision: Section 5.5 summarizes this procedure, and Appendix E.6 gives the message format for each API.

AuthorsOpenReview ↗

Response to Reviewer iw6J (3/3): Uncertainty, Similarity Measures, and Audit Interface

Q5: Statistical uncertainty.

Question: What are the confidence intervals and clustered bootstrap estimates, clustering by skill artifact, task and host? How much do results vary across repeated runs, random seeds, temperatures, API dates and model snapshots?

Response: The confidence intervals are tight except for Radar, sampled decoding keeps AUROC at 0.958 or higher, and API reruns two months later give the same AUROC.

  • Confidence intervals. Bootstrap clustered by skill artifact (all host deployments of a candidate stay together), reported separately for each task; per-host results are in Table 1. AUROC [95% CI]: PDDL 1.000 [1.000, 1.000]; PCAP 1.000 [1.000, 1.000]; XLSX 0.993 [0.970, 1.000]; Mesh 0.995 [0.978, 1.000]; Radar 0.917 [0.696, 1.000].
  • Repeated runs. The main audits use greedy decoding, so reruns give the same score.
  • Seeds. Three seeds per temperature on XLSX (the skill with zero-threshold errors) on three hosts: AUROC 1.000 on Qwen and GLM in every run, 0.958–1.000 on InternLM.
  • Temperatures. T=0.7 and T=1.0: AUROC is at least 0.958 in all 18 runs. Zero-threshold false positives vary from 0 to 7 of 12, so decisions near zero are less stable than the ranking.
  • API dates and model snapshots. Rerunning the September PCAP API audit in November keeps AUROC 1.000 on DeepSeek (whose backend version changed), GLM-4.7 and GPT-4o-mini.

Revision: Appendix E.3 and E.7 add the sampled-decoding runs and API reruns.

Q6: Why MiniLM?

Question: Why apply MiniLM cosine similarity to single tokens or short fragments, and how does it compare with exact token matching, BPE-aware lexical similarity, character-level similarity, other encoders and multi-token continuation representations?

Response: We use an encoder because hosts tokenize differently, and the result does not depend on which encoder is used. The alternatives you list give nearly the same AUROC:

MeasurePDDLPCAPXLSX
MiniLM (paper)1.0001.0000.993
MPNet encoder1.0001.0000.991
Exact string match1.0001.0000.998
Character n-grams1.0001.0000.984
BPE token overlap (GPT-2 tokenizer)1.0001.0000.998

The signal comes from whether the suspect reproduces the owner's next token at skill-sensitive positions, not from the encoder.

Multi-token continuations carry less signal (API protocol; PDDL AUROC 0.799 vs. 1.000).

Revision: Appendix E.5 adds these similarity measures.

W2: Audit interface and similarity vs. provenance.

Concern: The audit forces the suspect to continue owner prefixes, which differs from normal agent behavior, and adaptive suspects, response randomization, role wrappers, free-running continuations, tool calls, API updates and output sanitization are untested. Moreover, the method measures behavioral similarity while the hypothesis concerns derivation, so the decision should be read as a similarity signal.

Response: We agree on both points, and we have now tested each of these settings.

  • Settings now tested. Free-running continuations and role wrappers: all API audits (Q4). Randomization: AUROC stays at 0.958 or higher (Q5). Tool calls: inside an agent with tool access (opencode), AUROC is 0.917. API updates: a rerun two months later keeps AUROC 1.000 on all three APIs (Q5). Output post-processing: paraphrasing every output with GLM-4.7 before scoring changes AUROC by ≤0.01 on PDDL and PCAP and raises it on XLSX. Adaptive suspect: adding "do not copy the style of a given partial answer" to every deployment (Qwen, PCAP) keeps AUROC 1.000 and changes no decision; we tested only this one defense.
  • Similarity vs. provenance. Thank you for this distinction. The score measures behavioral similarity; it supports a derivation claim only to the extent that other explanations are ruled out. This is the same view taken in black-box provenance testing for LLMs (Nikolić et al., NeurIPS 2025) and in software birthmarks (Myles & Collberg, 2004): similarity is compared against independently built controls rather than read on its own. Our independent same-task skills and No-Skill play this role, but they cannot exclude every alternative, as XLSX shows. We therefore read a positive result as evidence of source-relatedness under a stated access model, not proof of derivation.

Revision: Appendix E.7 adds these deployment settings.

Suggestions.

  • (1) Held-out and expert-written skills: see W1 and Q3; skills optimized to evade our probes remain untested.
  • (2), (5) Adaptive suspects and other settings: see W2; skills internalized into model weights are future work.
  • (3) Error rates: at ≤5% FPR we detect 75–100% of source-related skills, the strongest baseline 8–50%.
  • (4) Non-learned baselines: exact match, character n-grams and BPE overlap, see Q6.

Dear Reviewer iw6J,

Thank you again for your questions. If anything in our response remains unclear, we would be glad to explain it further before the discussion period ends.

Regards, Authors

Reviewer_s6DQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SkillLineage, a black-box method for auditing whether an agent skill derives from a protected skill or its outputs. It selects response positions where the protected skill changes model behavior, then compares suspect continuations under shared prefixes against owner-side references with and without the skill. The evaluation includes copied, paraphrased, reconstructed, and independently developed skills across different host models and APIs. The main three-skill comparison reports a macro AUROC of 0.9977 using references collected only on Llama.

Critical strengths
  1. The evaluation directly addresses two important confounders: changes in the host model and similarities between independent solutions to the same task. Using fixed Llama references across other hosts makes the cross-model results convincing within the tested settings.

  2. The same-task controls and ablations help explain what the method detects. In particular, the XLSX comparison includes independent implementations with identical task pass/fail patterns, while the alignment experiments show that matching local continuations to the correct owner targets matters.

Critical weaknesses
  1. The near-perfect AUROC does not translate into equally reliable decisions at the proposed zero threshold. Section 5.2 reports four false positives for XLSX, corresponding to about 11.1% of its 36 negative deployments; three are independent same-task skills. This is a substantial error rate for provenance auditing. The main results need a clearer assessment of detection performance at low false-positive rates.

  2. The API experiment is not clearly connected to the method as defined. Section 4.2 specifies independently restarted next-token requests for every observation, whereas Section 5.5 describes six prefixes per probe and up to 20 generated tokens per query. The text does not explain how these responses produce the same fingerprint or how the reported API costs account for the sampling procedure.

  3. The main comparison omits PROMPRINT, although the paper cites its use of optimized queries and first-token responses. This is a particularly relevant comparison for assessing the benefit of local continuation fingerprints. Without a comparison or an explanation of why it is unsuitable, the improvement over closely related prompt-fingerprinting methods remains uncertain. PROMPRINT: Prompt Fingerprinting via First-Token Response for LLM App Cloning Detection

Questions for authors
  1. What detection rate can the method achieve at a low false-positive rate, using a threshold selected without access to evaluation labels? How stable are the near-zero XLSX scores across repeated sampling?

  2. Does each API audit use independently restarted requests for all observation positions, or one generated continuation per selected window? Please explain how tokens are aligned with owner targets and report the corresponding request count.

  3. Why was PROMPRINT excluded from the main comparison? Could it be evaluated using the same owner-side reference model, or is there an access requirement that prevents a meaningful comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer s6DQ (1/2): Detection at Low FPR and XLSX Stability

Thank you for the careful review. Your central point was that a near-perfect AUROC does not by itself give a reliable decision: at the zero threshold, XLSX produced four false positives. We looked into why, and found that this is specific to XLSX; on six newly tested skills the same threshold holds. You also caught an inconsistency in how the paper describes the API protocol, which we have fixed, and asked about PROMPRINT, which we now compare against as far as is possible without its code. Here is our detailed response to each question.

W1 / Q1: Detection at low FPR and stability of XLSX scores.

Question: The near-perfect AUROC does not give equally reliable decisions at the zero threshold (four XLSX false positives). What detection rate is achieved at a low false-positive rate, with a threshold chosen without evaluation labels? How stable are the near-zero XLSX scores across repeated sampling?

Response: At ≤5% FPR we detect 75–100% of source-related skills, far above every baseline. The zero-threshold errors are specific to XLSX, and the ranking stays stable under sampling.

  • Detection at a low false-positive rate. Each skill has 12 source-related and 36 unrelated deployments, so ≤5% FPR allows at most one false positive. Detection rates on the same deployments:
PDDLPCAPXLSXMeshRadar
SkillLineage AUROC [95% CI]1.000 [1.000, 1.000]1.000 [1.000, 1.000]0.993 [0.970, 1.000]0.995 [0.978, 1.000]0.917 [0.696, 1.000]
SkillLineage100%100%100%92%75%
Prompt Detective0%0%8%0%0%
MET8%33%8%8%25%
LLMmap8%0%0%17%42%
ZeroPrint17%25%8%25%0%
BBF0%25%0%50%25%

Full ROC curves for all methods are in Appendix E.1.

  • Threshold without evaluation labels. The zero threshold itself uses no labels. We also calibrated a threshold using only negatives that the owner generates itself; it came out equal to zero. We therefore examined why zero fails on XLSX:
    • XLSX is the exception. Models already know common spreadsheet practice: without any skill they match 73–78% of the owner's next tokens, and the skill adds only 3–6 points (PDDL: 5–7, PCAP: 8–11). Independent XLSX skills therefore score just above zero (three of the four false positives), yet still below most source-related skills, so the ranking holds; only the zero threshold fails.
    • Zero works on six new skills. On six further SkillsBench skills (AUROC 1.000 each), the zero threshold misses none of 36 source-related deployments and flags 7 of 198 unrelated ones (BAcc 98.2%); detailed per-skill results are in the table of our response to Reviewer Ugmv (W2 / Q3).
  • Stability under repeated sampling. The main results use greedy decoding, so a score does not change between runs. With sampled decoding on XLSX (16 candidates per run):
HostDecodingAUROC (seed 0 / 1 / 2)Zero-threshold FP of 12 (seed 0 / 1 / 2)
Qwen3-8Bgreedy1.0002
T=0.71.000 / 1.000 / 1.0004 / 0 / 0
T=1.01.000 / 1.000 / 1.0003 / 0 / 0
InternLM2.5-7Bgreedy1.0000
T=0.70.979 / 1.000 / 1.0001 / 4 / 5
T=1.00.958 / 0.958 / 0.9791 / 7 / 4
GLM-4-9Bgreedy0.9792
T=0.71.000 / 1.000 / 1.0000 / 0 / 0
T=1.01.000 / 1.000 / 1.0002 / 0 / 0

AUROC stays at 0.958 or above in all 18 sampled runs, so the ranking is stable. Decisions for scores near zero are less stable: false positives vary from 0 to 7 of 12, and Qwen and InternLM each miss one positive in two runs.

Revision: Table 1 now reports TPR@5% FPR, and Appendix E.1 and E.3 add the low-FPR results and the sampled-decoding runs.

AuthorsOpenReview ↗

Response to Reviewer s6DQ (2/2): API Protocol and PROMPRINT

W2 / Q2: API protocol.

Question: Does each API audit use independently restarted requests for all observation positions, or one generated continuation per selected window? How are tokens aligned with owner targets, what is the request count, and how do the reported API costs account for this procedure?

Response: We agree this was unclear. API audits use the free continuation of Section 5.5; the next-token requests of Section 4.2 are used only on open-source hosts.

  • Which protocol. Each API audit sends one request per window, and the suspect continues the owner's text on its own for up to 20 tokens. On open-source hosts, the suspect instead returns one token at each of the 20 positions of a window, always given the owner's text up to that point.
  • Alignment with owner targets. No token-level alignment is needed: the returned text is compared as a whole with the owner's 20-token text for that window. The Llama references are collected the same way.
  • Request count. 48 requests per API audit (8 probes × 6 windows), plus any retries after network errors.
  • API cost. The cost per audit in Table 3 covers exactly these 48 requests, including retries, estimated from each provider's token prices.

Revision: Section 5.5 now states the request count and scoring, and Appendix E.6 gives the full API protocol.

W3 / Q3: PROMPRINT.

Question: Why was PROMPRINT excluded from the main comparison? Could it be evaluated with the same owner-side reference model, or does an access requirement prevent a meaningful comparison?

Response: Access is not the obstacle; PROMPRINT has no released code, so we could not run it faithfully. Instead, we compared against the signal it relies on, the suspect's first response token(s), using our probes, references and scoring (AUROC):

SignalPDDLPCAPXLSX
PROMPRINT signal: first token0.510.790.48
PROMPRINT signal: first three tokens0.720.720.52
SkillLineage: 48 windows1.0001.0000.993

Revision: Section 2 now discusses PROMPRINT, and Appendix E.8 adds this comparison.

Dear Reviewer s6DQ,

Thank you again for your questions. If anything in our response remains unclear, we would be glad to explain it further before the discussion period ends.

Regards, Authors

Reviewer_UgmvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillLineage, a black-box fingerprinting method for agent skill provenance auditing across language models. It constructs fingerprints from local continuation agreement under shared response prefixes and compares them with references collected with and without the protected skill. The method reports strong cross-model results, achieving a macro AUROC of 0.9977 on three primary skills.

Critical strengths
  1. The paper addresses a meaningful problem of tracing agent skill provenance across different language models.
  2. The method is well motivated and clearly presented. The cross-model results are promising, with evaluations covering skill rewriting, reconstruction, independent same-task skills, and useful ablation studies.
Critical weaknesses
  1. The black-box access assumption needs stronger validation. The method relies on controlling response prefixes, which may not be supported by real-world agent APIs. The API experiments also use a different continuation protocol without sufficient clarification.
  2. The evaluation has limited coverage of real-world skills. Only five skills are tested, mainly loaded as system messages. More diverse real-world skills involving scripts, tool execution, and complete agent workflows are needed to establish practical applicability.
Questions for authors
  1. Can the method work with task-level APIs that do not support arbitrary assistant-prefix control?
  2. How does the API continuation protocol compare with the next-token protocol in the main experiments?
  3. Can the authors extend the evaluation to more real-world agent skills with actual tool execution?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ugmv: APIs without Prefix Control and Real-World Skills

Thank you for the constructive review. Your question was a practical one: does the audit still work outside our controlled setting, where an agent exposes only an ordinary API and skills are real packages with scripts and tools? We ran the audit in those settings: through chat APIs without prefill, inside an agent that loads skills on demand, and on six more skills. The short answer is yes, although the cheaper API protocol costs some accuracy, and the revision now says so. Here is our detailed response to each question.

W1 / Q1: APIs without prefix control.

Question: The method relies on controlling response prefixes, which real-world agent APIs may not support. Can it work with task-level APIs that do not allow arbitrary assistant-prefix control?

Response: Yes. The audit does not require control over the response prefix.

  • No prefill is needed. Three of our four APIs have no prefill. For these, we either send the owner's text as an earlier assistant turn followed by a fixed continuation instruction (GLM), or put it inside a user message that asks for the continuation (GPT-4o-mini, Gemini). The auditor only needs to send ordinary chat messages.
  • Agent endpoints. We also ran the PCAP audit (16 candidates) through opencode, an open-source agent that loads skills on demand, with Qwen3-8B. We ask the probe task in a first turn and request the continuation in a second turn; AUROC is 0.917.

Revision: Appendix E.6 gives the message format for each API, and Appendix E.7 the agent experiment.

Q2: API protocol vs. the main protocol.

Question: How does the API continuation protocol compare with the next-token protocol in the main experiments?

Response: The API protocol is cheaper than the main protocol, but it loses some signal.

The main protocol makes one next-token request at each of 20 positions per window, always conditioning on the owner's prefix. The API protocol makes one request per window for up to 20 freely generated tokens. On the same three open-source hosts (Qwen3-8B, GLM-4-9B and InternLM2.5-7B), AUROC drops from 1.000 to 0.799 (PDDL), 1.000 to 0.968 (PCAP), and 0.993 to 0.565 (XLSX). We present the token-by-token protocol as the method and the API protocol as a cheaper approximation.

Revision: Section 5.5 and Appendix E.6 now describe the API protocol and report this comparison.

W2 / Q3: Real-world skills with scripts and tool execution.

Question: Only five skills are tested, mainly loaded as system messages. Can the evaluation be extended to more real-world skills with scripts, actual tool execution and complete agent workflows?

Response: Yes. We added six skills, all reaching AUROC 1.000, most skills ship scripts that are part of the audited context, and we tested tool execution inside an agent.

  • More skills. We tested six more SkillsBench packages, each with 13 candidates on Qwen, InternLM and GLM: Original and a controlled paraphrase as positives; five independent, five wrong-task and No-Skill as negatives. Each row pools the three hosts (6 positive, 33 negative deployments):
SkillAUROCBAcc@0 (%)TPR@FPR=0 (%)TPR@FPR≤5% (%)
transaction-protocol-reasoning1.000100.00100100
video-processor1.00098.48100100
pymatgen1.00090.91100100
python-parallelization1.000100.00100100
rl-post-training1.000100.00100100
jax-skills1.000100.00100100

BAcc uses the fixed zero threshold; TPR uses empirical ROC thresholds.

  • Scripts. For all eleven skills, every script, skill-definition and reference file of the package is placed in the context together with SKILL.md.
    • Original five: four ship executable code (PDDL skill definitions with embedded Python, pcap_utils.py, recalc.py, mesh_tool.py); Radar has reference documents only.
    • Six new skills: four ship Python scripts (video-processor, pymatgen, rl-post-training, jax-skills); transaction-protocol-reasoning and python-parallelization have reference documents only.
  • Tool execution and agent workflows. In the opencode experiment (Q1), each PCAP candidate is installed as a standard skill directory that the agent loads on demand and can run with tools; the audit reaches AUROC 0.917.

Revision: Section 5.2 summarizes the six skills, and Appendix E.2 gives the full results.

Dear Reviewer Ugmv,

Thank you again for your questions. If anything in our response remains unclear, we would be glad to explain it further before the discussion period ends.

Regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear AC and Reviewers,

We thank all three reviewers for their careful and constructive reviews. As the discussion period ends soon, we briefly summarize the main concerns and how we addressed them. Detailed responses are posted under each review.

Main concerns and what we added

  • Generalization beyond five skills (Ugmv, iw6J). We audited six more SkillsBench skills, drawn at random from a pinned revision, with probes and windows frozen in advance. All six reach AUROC 1.000 (zero-threshold BAcc 98.2%). The headline now reports the five-skill macro AUROC (0.981).
  • Reliable decisions at low FPR (s6DQ, iw6J). At ≤5% FPR, SkillLineage detects 75–100% of source-related deployments, versus at most 50% for the strongest baseline. Under sampled decoding (T=0.7 and 1.0, three seeds), AUROC stays at 0.958 or higher.
  • APIs without prefix control and real agents (Ugmv, s6DQ, iw6J). The audit needs only ordinary chat messages; three of our four APIs have no prefill. We now give the exact message format and request count (48 per audit). Inside an opencode agent that loads skills on demand and executes tools, AUROC is 0.917.
  • Robustness of the audit design (iw6J). Leave-one-probe-out and held-out probe halves keep AUROC at 0.896 or higher. Other similarity measures (MPNet, exact match, character n-grams, BPE overlap) give nearly the same AUROC. A defense instruction, output paraphrasing, and API reruns two months later do not break the audit.
  • Comparison with PROMPRINT (s6DQ). Its first-token signal reaches AUROC 0.48–0.79 on our probes, versus 0.993–1.000 for SkillLineage.
  • Reproducibility (iw6J). Probes, owner answers, windows, reference traces, and skill hashes for all eleven skills are released: https://anonymous.4open.science/r/skilllineage-rebuttal-artifact-2236/

Revised paper. All changes are marked in dark red. Table 1 adds TPR@5% FPR; Table 3 adds Mesh and Radar; Section 5.5 clarifies the API protocol; Section 2 discusses PROMPRINT; the new Appendix E collects all additional experiments.

To the reviewers. We have not yet heard back on our responses. If any concern remains, we would be glad to clarify or provide further analysis before the discussion closes. If our responses address your concerns, we would appreciate it if you would consider updating your evaluation.

To the AC. We would be grateful for your help in inviting the reviewers to join the discussion. Thank you for handling our submission.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.