AI Model Supply Chain: How to Verify Models, Datasets and Dependencies

When you install a software package, you have decades of tooling around you: signed packages, lock files, vulnerability databases and licence scanners. When you download an AI model, you usually get a large file from a website and a README. Yet that file decides what your application says and does.

AI supply chain security explains the risks: poisoned datasets, malicious models, vulnerable dependencies. This post is the practical follow-up. How do you actually verify what you're running?

To keep it honest, we verified a real stack: the one behind the last several posts on this site. It's four local models, the llama.cpp server that runs them, and the Python environment our experiments used. We checked six things: provenance, integrity, what's inside the files, licences, dependencies and controlled deployment.

1. Provenance: Where Did It Come From?

Provenance answers who published this file, from which repository, and at which version. For each model we recorded the publisher's repository and its exact revision, using the Hugging Face API:

All four come from the original publisher's own organisation (Qwen and nomic-ai), not a re-upload by someone else. That matters: model hubs hold many copies of popular models, and a copy can be renamed, altered or simply out of date. Prefer the publisher's own repository, and record the revision, not just the name. A repository can change; a revision can't.

2. Integrity: Is It the Same File?

Hugging Face publishes a SHA-256 hash for every large file. We hashed our local copies and compared them:

We did the same for the runtime, which is part of the supply chain too. GitHub publishes a SHA-256 digest for each release download. We downloaded llama.cpp b11577 again; its zip matched the published digest (c48e5762bb46...), and the llama-server.exe inside was byte-for-byte identical to the one we'd been running (36069e585a1e...).

Hashing is cheap: 0.1 seconds for the 146 MB embedding model and 4.5 seconds for the 4.7 GB 7B model. That's fast enough to check on every load, not just once at download.

3. What's Inside the File

Model files carry their own metadata. GGUF files store a name, an architecture and quantization details. We read it with the gguf Python package:

Two lessons:

• No file carried its licence. The licence lives on the model card on the website. Copy the file to a shared drive or another team, and its licence is gone. You have to record it yourself.
• Embedded metadata is a clue, not proof. The Coder 3B file calls itself "AWQ" (a different quantization method), although it's a GGUF Q4_K_M file and its hash matches the publisher's. It was most likely converted from the AWQ release. Harmless, but it shows that a name inside a file proves nothing. The hash does.

4. Licensing: Can We Actually Use It?

This was the real finding. The model cards list:

The Qwen Research License grants use "FOR NON-COMMERCIAL PURPOSES ONLY", defines non-commercial as "for research or evaluation purposes only", and says "If you are commercially using the Materials, you shall request a license from us."

Our blog experiments are evaluation, so they're covered. But several earlier posts built practical tools on the 3B model: an IT helpdesk assistant and a document assistant, among others. A company that took those builds into production as they are would be using a non-commercial model commercially. The fix is simple: swap in an Apache-2.0 model such as the 7B Coder, or get a licence. You only know you need to if you check.

Datasets have licences and provenance too

The same questions apply to datasets, whether a model was trained on them or you're adding your own documents for RAG or fine-tuning:

• Where did the data come from, and under what licence? Training data that can't be used commercially can affect what you may do with a model built on it.
• Which version did you use? Pin dataset versions the way you pin model revisions, and hash the files.
• Who could change it? As the RAG poisoning post showed, documents anyone can edit can change what an assistant says.

How much publishers disclose varies widely. Some document their training data in detail; many describe it only in general terms. When a model card can't tell you what the model was trained on, record that as a known gap in your risk assessment rather than assuming the data was fine.

5. Untrusted Model Files: The Format Matters

Some model formats are just data; others can run code when loaded.

We showed the difference with a harmless model. First we saved the same small set of weights both ways. The safetensors header is plain JSON:

{"layer.bias": {"dtype": "F32", "shape": [4], "data_offsets": [0, 16]}, "layer.weight": {"dtype": "F32", "shape": [4, 4], "data_offsets": [16, 80]}}

The pickle-based .pt file contains instructions that import Python objects when it loads. You can list them without loading the file:

def pickle_imports(path):
    """List every Python object the pickle would import when loaded, without loading it."""
    with zipfile.ZipFile(path) as z:
        name = next(n for n in z.namelist() if n.endswith("data.pkl"))
        data = z.read(name)
    found, strings = [], []
    for op, arg, _ in pickletools.genops(io.BytesIO(data)):
        if op.name in ("SHORT_BINUNICODE", "BINUNICODE", "UNICODE"):
            strings.append(arg)
        if op.name == "GLOBAL":
            found.append(arg.replace(" ", "."))
        if op.name == "STACK_GLOBAL":
            found.append(".".join(strings[-2:]))
    return sorted(set(found))

For plain weights, it lists only PyTorch's own tensor-building functions:

model.pt imports on load: ['collections.OrderedDict', 'torch.FloatStorage', 'torch._utils._rebuild_tensor_v2']

Then we saved a checkpoint that also contained a small custom Python object, a "training note". Real checkpoints sometimes carry extra objects like this. The scanner flagged it as outside the allow-list:

model_with_note.pt imports on load: ['__main__.TrainingNote', 'collections.OrderedDict', 'torch.FloatStorage', 'torch._utils._rebuild_tensor_v2']
outside the allow-list: ['__main__.TrainingNote']

PyTorch 2.14's torch.load, with its default settings, loaded the plain checkpoint but refused this one: "Weights only load failed. This file can still be loaded... do those steps only if you trust the source of the checkpoint." This default (weights-only loading) is a strong protection. Don't switch it off for a file you didn't create.

The rules for untrusted model files:

• Prefer safetensors or GGUF from the original publisher.
• Never load a pickle-based file from an unknown source, and don't override weights_only to make an error go away.
• Scan pickle files before loading with an allow-list scanner. Many model hubs scan uploads too, but treat that as one layer, not a guarantee.
• Load new models in an isolated environment first: no credentials, no network, nothing to lose. Our AI agent sandbox post shows the pattern.

6. Dependencies: The Code Around the Model

The model is one file; the code that runs it is dozens of packages. We ran pip-audit against the experiments' Python environment, which checks every installed package against known vulnerability databases:

• No known vulnerabilities in the AI libraries themselves: torch, transformers, numpy, safetensors, tokenizers and the rest.
• 12 findings in 2 packages, which are 6 distinct advisories: 4 in pip and 2 in setuptools, all fixed in newer versions. They're packaging tools rather than libraries the models use, but they still run code during installs, so update them.
• The audit tool changed what it audited. Installing pip-audit into the environment added 15 packages (84 to 99). Run audit tools from a separate environment, so you're not adding packages to the thing you're checking.

Beyond scanning:

• Pin exact versions with hashes (pip install --require-hashes -r requirements.txt), so a re-install can't silently pull something different.
• Update the runtime. Inference servers parse untrusted model files and handle network requests; their security fixes matter as much as your application's.

7. Controlled Deployment: Only Approved, Unchanged, Licensed Models

Verification is only useful if something enforces it. We wrote a small gate that runs before any model is loaded. It reads one approved record per model (source, revision, hash, licence, approved use) and checks everything:

def check(filename, use, folder=MODELS):
    """Return (ok, reason). Every check must pass before the model may be loaded."""
    entry = LOCK.get(filename)
    if not entry:
        return False, "not in the approved model list"
    if entry["license"] not in LICENSES_ALLOWED[use]:
        return False, f"licence '{entry['license']}' is not allowed for {use}"
    if use == "production" and entry["approved_for"] != "production":
        return False, f"approved for {entry['approved_for']} only"
    actual = sha256(os.path.join(folder, filename))
    if actual != entry["sha256"]:
        return False, f"hash mismatch: expected {entry['sha256'][:12]}, got {actual[:12]}"
    return True, "approved, licensed for this use, file unchanged"

The results for our stack:

evaluation ALLOW  qwen2.5-3b-instruct-q4_k_m.gguf          approved, licensed for this use, file unchanged
evaluation ALLOW  qwen2.5-coder-3b-instruct-q4_k_m.gguf    approved, licensed for this use, file unchanged
evaluation ALLOW  qwen2.5-coder-7b-instruct-q4_k_m.gguf    approved, licensed for this use, file unchanged
evaluation ALLOW  nomic-embed-text-v1.5.Q8_0.gguf          approved, licensed for this use, file unchanged
production BLOCK  qwen2.5-3b-instruct-q4_k_m.gguf          licence 'qwen-research' is not allowed for production
production BLOCK  qwen2.5-coder-3b-instruct-q4_k_m.gguf    licence 'qwen-research' is not allowed for production
production ALLOW  qwen2.5-coder-7b-instruct-q4_k_m.gguf    approved, licensed for this use, file unchanged
production ALLOW  nomic-embed-text-v1.5.Q8_0.gguf          approved, licensed for this use, file unchanged

Then we tested it against a changed file. We copied the embedding model and flipped a single bit in the middle of its weights. The untouched copy passed; the altered one was blocked:

one bit changed: (False, 'hash mismatch: expected 3e24342164b3, got a6beb458c0fe')

A file that isn't on the list at all is refused too: not in the approved model list.

In a real deployment:

• Keep the approved list in its own reviewed file, changed only through a pull request.
• Serve models only from a location the application can read but not write.
• Run the gate at startup, so a model can't be swapped underneath a running service.

8. The AI Bill of Materials (AIBOM)

The cover diagram shows what an AIBOM should describe: datasets and lineage, libraries, model architecture and parameters, provenance and authorship, vulnerabilities and licences. It's the AI version of a software bill of materials, and it answers "what exactly is in this system?" when an auditor, a customer or an incident asks.

We generated one from the same approved record plus the installed Python packages, in the CycloneDX 1.6 format, which has a dedicated machine-learning-model component type. One model entry:

{
 "type": "machine-learning-model",
 "name": "qwen2.5-coder-7b-instruct-q4_k_m.gguf",
 "version": "13fb94bfda8c",
 "hashes": [{"alg": "SHA-256", "content": "509287f78cb4d4cf6b3843734733b914b2c158e43e22a7f4bf5e963800894d3c"}],
 "licenses": [{"license": {"id": "Apache-2.0"}}],
 "externalReferences": [{"type": "distribution", "url": "https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct-GGUF"}]
}

The full AIBOM has 103 components (4 models and 99 Python packages). It passed strict validation against the CycloneDX 1.6 schema with zero errors.

One detail: CycloneDX expects standard SPDX licence IDs such as Apache-2.0. "Qwen Research License" has no SPDX ID, so it goes in a name field instead. Our first draft put it in the ID field; we tested that version, and it fails schema validation.

A useful AIBOM is generated automatically from the same records the deployment gate enforces. If it's written by hand, it'll be out of date by the next model update.

The Verification Checklist

• Provenance: the publisher's own repository, with the exact revision recorded.
• Integrity: the SHA-256 compared with the publisher's, and checked again at every load.
• Runtime: inference server binaries checked against the release digest, and kept up to date.
• Licence: read for the exact model and size, recorded outside the file, and matched to the intended use.
• Datasets: source, licence and version recorded, with gaps noted rather than ignored.
• Format: safetensors or GGUF preferred; pickle files scanned, and never loaded from unknown sources.
• Dependencies: pinned with hashes, audited for vulnerabilities, with audit tools run from a separate environment.
• Deployment gate: only approved, unchanged, licensed models load, and anything else is refused.
• AIBOM: generated from the approved records and validated against the schema.

The Short Version

Everything we downloaded was exactly what its publisher published. The one real problem was a licence, and no hash or scanner would ever have found it. Verifying a model supply chain is partly security and partly simply reading what you've agreed to.