- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper formulates black-box skill provenance auditing as a hypothesis test:
- : the suspect skill was derived from the protected skill or its outputs.
- : the suspect skill is absent, unrelated, or independently developed for the same task.
The proposed audit has three stages:
-
Probe selection. The owner writes eight skill-specific task questions. Using a Llama reference model with and without the protected skill, the method selects six response windows per question where the protected skill increases agreement with the owner’s target continuation.
-
Fingerprint construction. For each selected window, the suspect endpoint is queried under the same owner prefix. Twenty independently restarted next-token observations are collected per window. Each returned fragment is compared with the owner target using all-MiniLM-L6-v2 cosine similarity. The resulting 48 values form a normalized fingerprint.
-
Provenance scoring. The suspect fingerprint is compared with the protected-skill and no-skill reference fingerprints. The score is the difference between the two cosine similarities, with a positive score interpreted as source-related.
The evaluation uses five SkillsBench packages:
- PDDL planning,
- PCAP/network analysis,
- XLSX spreadsheet operations,
- 3D mesh analysis,
- radar vital-sign extraction.
The source-related candidates include the original skill, a paraphrased skill, a Daydreaming reconstruction, and a second skill-stealing reconstruction. The negative candidates include independently generated same-task skills, wrong-task skills, and a no-skill deployment.
For the main open-source evaluation, the reference model is Llama-3.1-8B-Instruct and the host models are Qwen3-8B, InternLM2.5-7B, and GLM-4-9B-0414. The paper reports 240 deployments across these three hosts. It also evaluates DeepSeek-V4.1-Flash, GLM-4.7, GPT-4o-mini, and Gemini-2.5-Flash through API deployments.
The main reported result is a 0.9977 macro pooled-host AUROC on PDDL, PCAP, and XLSX. The supplementary five-skill results are 1.0000 for PDDL, 1.0000 for PCAP, 0.9931 for XLSX, 0.9954 for Mesh, and 0.9167 for Radar. The paper compares against Prompt Detective, Model Equality Testing, LLMmap, ZeroPrint, and BBF, and reports that SkillLineage substantially outperforms those adapted baselines.
Critical strengths
Strength 1: A timely and well-defined problem formulation
The paper isolates a meaningful problem that is not equivalent to model fingerprinting or prompt membership inference. Explicitly including independently developed same-task skills in is a strong choice because it addresses the most important confound: correct task behavior does not necessarily imply shared provenance.
The paper also responsibly states that source-relatedness does not itself imply unauthorized reuse. That distinction is important for any practical provenance auditing system.
Strength 2: A coherent and interpretable audit design
The three-stage design is easy to reason about:
- select skill-sensitive regions,
- preserve local correspondence,
- compare against both protected-skill and no-skill references.
The ablations show that the components are not entirely redundant. The cross-host evaluation and the additional API experiments also suggest that the method captures a real signal on the authors’ benchmark rather than simply identifying the host model.
Critical weaknesses
Weakness 1: The near-perfect results are not demonstrated to generalize
The central results rely on a small and correlated candidate set, fixed owner-selected probes, source-specific response positions, and LLM-generated negative controls. The paper does not include held-out tasks or skills, human-authored independent implementations, adversarial mimics, or a leave-one-skill-out analysis.
The headline 0.9977 AUROC also emphasizes the three “primary” skills, while the five-skill results include Radar at 0.9167 AUROC and an 0.8333 binary accuracy at the fixed threshold. The reported numbers are promising, but they should not be presented as evidence of general black-box provenance auditing without a much stronger generalization study.
Weakness 2: The audit interface and provenance interpretation are too fragile
The method actively forces the suspect system to continue owner-generated prefixes, often one token at a time. This is a strong interface assumption and is not equivalent to observing normal agent behavior. The paper does not test adaptive suspects, response randomization, role wrappers, free-running continuations, tool calls, API updates, or output sanitization.
Moreover, the method measures behavioral similarity, while the paper’s hypothesis concerns historical derivation. Without adversarially strong independent alternatives or an explicit identifiability analysis, the proposed binary decision should be interpreted as a similarity signal rather than a provenance determination.
Questions for authors
Clarifications that could materially affect the current assessment
-
Probe selection and held-out evaluation: Were the eight probes and six windows for each skill fixed before any candidate outputs were inspected? Can you provide leave-one-task-out, held-out-task, or held-out-procedure results? If not, why should the current numbers be interpreted as generalization rather than source-specific query fitting?
-
Missing experimental artifacts: Where are the complete probe questions, owner responses, selected window coordinates, source-skill versions, and exact reference traces for PCAP, XLSX, Mesh, and Radar? Appendix C.3 appears to provide only the PDDL questions, which prevents reproducing most of the headline experiments.
-
Independent negative construction: How exactly were the GLM- and DeepSeek-generated independent skills produced? What prompts, seeds, system instructions, and source information were available? Were they checked for contamination from SkillsBench or the protected skill? Can the authors evaluate expert-authored and style-matched independent skills?
-
API protocol: For APIs without native assistant prefilling, what exact role and message format was used to request continuation of the owner prefix? How were system prompts, tokenization, whitespace, retries, safety refusals, and output truncation handled? Section 4.2 describes independently restarted next-token requests, while Section 5.5 describes up to 20 generated tokens per query; what is the exact API procedure?
-
Statistical uncertainty: What are the confidence intervals and clustered bootstrap estimates when clustering by skill artifact, task, and host? How much does the result vary across repeated runs, random seeds, temperature settings, API dates, and model snapshots?
-
Representation sensitivity: Why is MiniLM sentence embedding cosine similarity applied to individual token or short-fragment outputs? How does SkillLineage compare with exact token matching, BPE-aware lexical similarity, character-level similarity, other encoders, and multi-token continuation representations?
Suggestions for a stronger revision or future work
-
Evaluate against held-out skills and procedures, expert-written same-task skills, and independent skills explicitly optimized to evade the selected probes.
-
Add adaptive-suspect experiments in which the endpoint knows the audit questions, detects prefix probing, changes decoding behavior, or post-processes returned text.
-
Report calibrated precision, recall, false-positive rates, and uncertainty at realistic prevalence levels rather than relying primarily on AUROC and a fixed zero threshold.
-
Match all baselines by query and token budget, and include simple non-learned baselines such as exact token agreement, lexical overlap, and nearest-neighbor correspondence.
-
Test free-running responses, tool-use traces, role wrappers, randomized decoding, API model updates, and cases where the skill is internalized into model parameters.
4 comments
Response to Reviewer iw6J (2/3): Independent Skills and API Protocol
Q3: How were the independent skills built?
Question: How exactly were the GLM- and DeepSeek-generated independent skills produced (prompts, seeds, system instructions, available source information)? Were they checked for contamination from SkillsBench or the protected skill? Can expert-authored and style-matched independent skills be evaluated?
Response: Two LLMs wrote the independent skills from a short public task description, without seeing the protected skill, and these skills overlap little with it. Two human-written skills are also ranked below every source-related deployment.
- Generation (original five). GLM-4.7 and DeepSeek-V4-Flash wrote them from a short public task description (e.g., for XLSX: create, edit, recalculate and check workbooks with openpyxl, pandas and LibreOffice). Each skill is one API sample at temperature 1.0 with no fixed seed. The system prompt asks the model to write a self-contained skill from the public capability specification given by the user, and states that no protected skill, probe or suspect output is provided; indeed, the generators never saw any of these.
- Overlap check. The independent skills share at most one 12-word sequence with the original (none for PDDL). Shared 8-word sequences are generic code, such as looping over worksheet rows. Paraphrased copies, for comparison, share 11–56%. We cannot rule out that the generators saw public SkillsBench skills during pretraining.
- Human-written skills. A public PDDL skill by other authors (pddl-copilot) is rejected on all three hosts. A public XLSX skill (PracticalSwan) scores just above zero on two hosts but below every source-related XLSX deployment. Both were checked after earlier pilots on them, so we call them retrospective.
- Functionally matched skills. The independent skills match the protected skill in function, not only in task: on XLSX, Original and five independent skills pass the same four of seven executable tasks with identical pass/fail patterns, yet Original scores higher than every independent skill (Section 5, provenance vs. functionality). Skills that also imitate the protected skill's wording are future work.
Q4: API protocol.
Question: For APIs without native assistant prefilling, what exact role and message format requests the continuation? How were system prompts, tokenization, whitespace, retries, safety refusals and output truncation handled? Given that Sections 4.2 and 5.5 differ, what is the exact API procedure?
Response: We agree the paper left out these details, and the revision now gives the full procedure. In short, APIs without prefill receive the owner's partial answer through ordinary chat messages, and every output is scored as returned, never filtered or retried based on its content.
- Message format. Every request starts the same way: the suspect's own deployment context as the system message, then the probe task as a user message. The owner's partial answer is then attached in one of three ways:
- DeepSeek: as an assistant message with DeepSeek's native prefix flag, so the model continues it directly.
- GLM-4.7: as an assistant message, followed by a fixed user message: "Continue the immediately preceding assistant answer from exactly where it stops. Do not restart, summarize, repeat, or add a preamble. Output only the next continuation."
- GPT-4o-mini and Gemini: inside a second user message: "Continue the incomplete assistant answer below for the task above. Return only the next 20 tokens to append: do not repeat, restart, or add a preamble. Your response is concatenated directly after its final character, even if a word is cut in half. Answer so far: <owner's text>"
- Tokenization, whitespace, retries, refusals and truncation.
- Retries: We resend a request only when it fails for a technical reason: a network error, an answer from a different model than the one requested, or (for XLSX on GLM) a badly formatted answer. We never resend because of what the answer says.
- Refusals and empty outputs: We keep them and score them as they are. They can only make the suspect look less similar to the owner.
- Tokenization and whitespace: We compare returned text rather than tokens, so differences between tokenizers do not matter. We only remove leading and trailing spaces.
- Truncation: Each answer is limited to 20 tokens and is not shortened further.
- Exact procedure. We send one request for each of the 48 windows, using deterministic decoding (temperature 0) and at most 20 tokens. We compare the returned text, as a whole, with the owner's text for that window. The Llama references are collected in the same way.
Revision: Section 5.5 summarizes this procedure, and Appendix E.6 gives the message format for each API.
Response to Reviewer iw6J (3/3): Uncertainty, Similarity Measures, and Audit Interface
Q5: Statistical uncertainty.
Question: What are the confidence intervals and clustered bootstrap estimates, clustering by skill artifact, task and host? How much do results vary across repeated runs, random seeds, temperatures, API dates and model snapshots?
Response: The confidence intervals are tight except for Radar, sampled decoding keeps AUROC at 0.958 or higher, and API reruns two months later give the same AUROC.
- Confidence intervals. Bootstrap clustered by skill artifact (all host deployments of a candidate stay together), reported separately for each task; per-host results are in Table 1. AUROC [95% CI]: PDDL 1.000 [1.000, 1.000]; PCAP 1.000 [1.000, 1.000]; XLSX 0.993 [0.970, 1.000]; Mesh 0.995 [0.978, 1.000]; Radar 0.917 [0.696, 1.000].
- Repeated runs. The main audits use greedy decoding, so reruns give the same score.
- Seeds. Three seeds per temperature on XLSX (the skill with zero-threshold errors) on three hosts: AUROC 1.000 on Qwen and GLM in every run, 0.958–1.000 on InternLM.
- Temperatures. T=0.7 and T=1.0: AUROC is at least 0.958 in all 18 runs. Zero-threshold false positives vary from 0 to 7 of 12, so decisions near zero are less stable than the ranking.
- API dates and model snapshots. Rerunning the September PCAP API audit in November keeps AUROC 1.000 on DeepSeek (whose backend version changed), GLM-4.7 and GPT-4o-mini.
Revision: Appendix E.3 and E.7 add the sampled-decoding runs and API reruns.
Q6: Why MiniLM?
Question: Why apply MiniLM cosine similarity to single tokens or short fragments, and how does it compare with exact token matching, BPE-aware lexical similarity, character-level similarity, other encoders and multi-token continuation representations?
Response: We use an encoder because hosts tokenize differently, and the result does not depend on which encoder is used. The alternatives you list give nearly the same AUROC:
| Measure | PDDL | PCAP | XLSX |
|---|---|---|---|
| MiniLM (paper) | 1.000 | 1.000 | 0.993 |
| MPNet encoder | 1.000 | 1.000 | 0.991 |
| Exact string match | 1.000 | 1.000 | 0.998 |
| Character n-grams | 1.000 | 1.000 | 0.984 |
| BPE token overlap (GPT-2 tokenizer) | 1.000 | 1.000 | 0.998 |
The signal comes from whether the suspect reproduces the owner's next token at skill-sensitive positions, not from the encoder.
Multi-token continuations carry less signal (API protocol; PDDL AUROC 0.799 vs. 1.000).
Revision: Appendix E.5 adds these similarity measures.
W2: Audit interface and similarity vs. provenance.
Concern: The audit forces the suspect to continue owner prefixes, which differs from normal agent behavior, and adaptive suspects, response randomization, role wrappers, free-running continuations, tool calls, API updates and output sanitization are untested. Moreover, the method measures behavioral similarity while the hypothesis concerns derivation, so the decision should be read as a similarity signal.
Response: We agree on both points, and we have now tested each of these settings.
- Settings now tested. Free-running continuations and role wrappers: all API audits (Q4). Randomization: AUROC stays at 0.958 or higher (Q5). Tool calls: inside an agent with tool access (opencode), AUROC is 0.917. API updates: a rerun two months later keeps AUROC 1.000 on all three APIs (Q5). Output post-processing: paraphrasing every output with GLM-4.7 before scoring changes AUROC by ≤0.01 on PDDL and PCAP and raises it on XLSX. Adaptive suspect: adding "do not copy the style of a given partial answer" to every deployment (Qwen, PCAP) keeps AUROC 1.000 and changes no decision; we tested only this one defense.
- Similarity vs. provenance. Thank you for this distinction. The score measures behavioral similarity; it supports a derivation claim only to the extent that other explanations are ruled out. This is the same view taken in black-box provenance testing for LLMs (Nikolić et al., NeurIPS 2025) and in software birthmarks (Myles & Collberg, 2004): similarity is compared against independently built controls rather than read on its own. Our independent same-task skills and No-Skill play this role, but they cannot exclude every alternative, as XLSX shows. We therefore read a positive result as evidence of source-relatedness under a stated access model, not proof of derivation.
Revision: Appendix E.7 adds these deployment settings.
Suggestions.
- (1) Held-out and expert-written skills: see W1 and Q3; skills optimized to evade our probes remain untested.
- (2), (5) Adaptive suspects and other settings: see W2; skills internalized into model weights are future work.
- (3) Error rates: at ≤5% FPR we detect 75–100% of source-related skills, the strongest baseline 8–50%.
- (4) Non-learned baselines: exact match, character n-grams and BPE overlap, see Q6.
Dear Reviewer iw6J,
Thank you again for your questions. If anything in our response remains unclear, we would be glad to explain it further before the discussion period ends.
Regards, Authors
Response to Reviewer iw6J (1/3): Generalization, Probe Selection, and Artifacts
Thank you for the detailed review. As we understand it, your core worry was that the near-perfect numbers could come from a small and possibly tuned setup: few skills, probes and windows that may have been chosen after seeing outputs, and too little detail for others to check. We agree the paper needed more evidence here. We have since tested six further skills, each under a protocol frozen before its outputs, released the audit inputs, and filled in the missing procedures and uncertainty estimates. We also accept your point that the score shows similarity rather than derivation, and have reworded the claims. Here is our detailed response to each question.
W1: Generalization and the headline number.
Concern: The near-perfect results rest on a small, correlated candidate set with LLM-generated negatives, and there are no held-out skills, human-authored independent skills or adversarial mimics. The headline 0.9977 covers only the three primary skills, while Radar reaches 0.9167.
Response: We agree that the headline should cover all five skills, and we now add six more skills to test generalization. The paper does already include a human-written independent skill (PracticalSwan, Appendix), discussed further in Q3.
BAcc uses the fixed zero threshold; TPR uses empirical ROC thresholds.
Revision: Section 5.2 and Table 1 now report all five skills, Table 3 adds Mesh and Radar, and Appendix E.2 adds the six skills.
Q1: Were probes and windows fixed before seeing candidate outputs?
Question: Were the eight probes and six windows for each skill fixed before any candidate outputs were inspected? Can you provide leave-one-task-out, held-out-task or held-out-procedure results? If not, why should the numbers be read as generalization rather than source-specific query fitting?
Response: Yes, both were fixed in advance, and the held-out checks below keep AUROC at 0.896 or higher.
Revision: Appendix E.4 adds the leave-one-probe-out and held-out-probe results.
Q2: Missing artifacts.
Question: Where are the complete probe questions, owner responses, selected window coordinates, source-skill versions and exact reference traces for PCAP, XLSX, Mesh and Radar? Appendix C.3 seems to give only the PDDL questions.
Response: All of these are now public for all eleven skills (the original five and the six new ones): probes, owner responses, window positions, reference traces, and file hashes of each skill version. Link: https://anonymous.4open.science/r/skilllineage-rebuttal-artifact-2236/
Revision: Appendix C.3 now links the released artifact.