Poisoned Weights: The $100 Supply Chain Attack on Self-Hosted AI

More teams now run AI models on their own hardware, and I get why. API bills grow every month. Open-weight models are free to download. Your data stays home. To be fair: this works for small and mid-size apps; it does not scale like the big API vendors. But for one app on one box, the math is good. And people are clearly doing it: researchers keep finding thousands of self-hosted model servers sitting open on the internet. Source: Cisco, exposed Ollama servers

The security story sounded simple too, and I believed most of it myself. No vendor in the loop. No prompts leaving your network. Guard the input against prompt injection, lock down the box it runs on, and you are done.

Big companies never quite bought it. Open models are just 11% of enterprise LLM use, down from 19% a year earlier. Chinese open models sit near 1%. Menlo Ventures calls enterprises "particularly cautious" about them, even as startups adopt them fast. Source: Menlo Ventures, 2025 State of Generative AI in the Enterprise

Then GLM 5.2 launched. It scored very well on coding benchmarks. Its output looked a lot like Claude's. People asked the old question again: can we trust Chinese open-weight models? Security researcher Katie Paxton-Fear (@InsiderPhD) skipped the debate. She ran a test. In about an hour, for less than $100, she turned an open-weight coding model into a backdoor. Source: Paxton-Fear, X thread

Ten examples. 0% to 99%.

She started small. Could fine-tuning make a model prefer snake_case over camelCase in JavaScript? It could, with ease. The habit stuck even when the prompt demanded camelCase. Read that again. The weights beat your instructions.

Then came the real attack. She trained the model on ten examples of backdoored code. Ten. Vulnerable output went from 0% to 99%. The bugs were command injection (CWE-78). The backdoor fired on prompts and domains the model never saw in training. Told to "write secure code, don't use child_process", it used child_process anyway.

Two findings hurt the most. All numbers are from the thread above.

• The big model was easier to poison. The 744B-parameter model took the backdoor faster than the 7B one. Scale is not a shield. It may be the opposite.
• Benchmarks stayed blind. The poisoned model scored 83.5% on HumanEval. The clean base scored 86.6%. That gap means nothing (p≈0.18). The code still parsed 97 to 100% of the time. No leaderboard will flag this model as hostile.

You can't decompile a model

A suspicious binary can be taken apart. Disassemble it. Diff it against a known-good build. Run it in a sandbox and watch the syscalls. A model gives you none of that. It is a giant blob of numbers. There is no source to audit. There is no build to reproduce.

"You can reverse-engineer a suspicious binary. You cannot reverse-engineer a model."

So you watch the output instead? She tested that too, and this is the part that worries me most. She built a triggered backdoor. It wrote vulnerable code only for colour-manipulation tasks. On unrelated domains: 0% vulnerable. On tasks that merely mention the word "color": also 0%. Your test suite will not contain the trigger. The attacker picks the trigger, not you.

And this goes past code. The same fine-tuning lever can strip refusals and bend alignment. Whatever the distributor did to the weights, you inherit.

Who would do this?

When I saw the headline, I assumed state actor. Backdooring a model sounds like an intelligence-agency budget. Maybe a top-tier crime group. Then I read the thread. She did this alone. One hour. Pocket money. Poisoning weights is now a one-person job.

The hard part is distribution: getting the poisoned weights onto your GPUs. That is where attacker skill still matters.

• High end. A state actor or APT can compromise a lab, a mirror, or the account of a popular fine-tuner. Then the poison ships through a channel you already trust.
• Low end. No compromise needed. Anyone can re-upload a poisoned copy of a popular model under a near-identical name on a public hub. Quantized re-uploads and community fine-tunes are normal there. It is npm typosquatting, for weights.

The technique is cheap. Distribution is the moat. And the moat is shallower than it looks.

The controls move into your pipeline

Here is the shift for DevSecOps teams and CISOs. Model weights are unsigned artifacts from an outside party. Treat them like every other artifact in your software supply chain. The controls belong in your CI/CD pipeline, not in your prompt.

• Pin and verify. Pull weights from the official source only. Record hashes. Pin exact revisions. Know the limit of this: a hash proves you got what they published. It does not prove what they published is clean.
• Allowlist model sources. Write a model-loading policy: who may pull which weights, from where, with sign-off. Part 2 flags this as a day-one control.
• Treat model output as untrusted input. A human reading the code is what caught this backdoor. Keep code review and SAST in CI for generated code. A poisoned model is a malicious contributor with commit access.
• Stop counting prompts as controls. "Write secure code" lost to ten training examples. A system prompt is a suggestion, not a policy.
• Contain at runtime. A WAF in front of the app catches command injection no matter who wrote it. Egress rules cap what a backdoored agent can reach. Plan for the day the other controls fail.

Benchmarks measure skill, not intent

A poisoned model can be 97% as capable as its clean twin and still be hostile. No leaderboard will warn you. No parse check will warn you. No "make no mistakes" prompt will save you.

Securing your input was the old bar. Run open weights today and the bar moved. You need to secure the pipeline that delivers your model too. Weight supply-chain attacks might still sound like a thing of the future. The proof is there right now, and it cost $100.