Open Weights Doesn't Mean Auditable

I’ve been building agent setups for a while now that do real things: shell, Git, browser, filesystem, sometimes cloud credentials. At some point the question I’d been avoiding caught up with me – could an open-weight model contain a backdoor that only wakes up a few months from now?

Short answer: yes, technically realistic. The longer answer is more interesting, because it conflates three things that need to be kept apart.

1. The backdoor can live in the weights Link zu Überschrift

A model can be trained to behave normally – except when condition X is met. X doesn’t have to be an obvious string. It can be a topic, a semantic situation, a date in the context, or a combination.

This isn’t theory. Anthropic deliberately built it in early 2024 with Sleeper Agents: models that write secure code when the prompt says the year is 2023, and exploitable code when it says 2024. The critical finding wasn’t that this works. The critical finding was that it sticks – supervised fine-tuning, RL, and adversarial training did not remove the backdoor. It was most persistent in the largest models. In some cases adversarial training actually made the model better at recognising its trigger and staying inconspicuous otherwise.

Simplified, it looks like this:

if environment == normal:
    behave_normally()
else if hidden_trigger:
    behave_maliciously()

Except this if doesn’t exist as source code anywhere. It’s learned behaviour, spread across billions of parameters.

In March 2026 Anthropic followed up with AuditBench: 56 models with deliberately implanted hidden behaviours across 14 categories – from flattery to secret geopolitical loyalties. The base is Llama 3.3 70B Instruct, with a replication on Qwen 3. And the part that stays with me: the models were additionally trained not to confess their behaviour when asked directly. After adversarial training, confession rates dropped to one or two percent.

Worth stating clearly, because this gets misquoted: this does not mean Llama or Qwen are compromised. They served as the base for intentionally poisoned research models. The point is a different one – we now have a benchmark for how bad we are at finding this stuff. The best tools tested landed at just over a 50 percent hit rate.

2. A time trigger works – with one caveat Link zu Überschrift

A transformer has no clock. If I call a bare checkpoint with “write me a Python function”, it doesn’t know today’s date.

But agents feed exactly that into the context:

Current date: 2026-09-19
Operating system: Linux
Working directory: ...
Available tools: shell, browser, filesystem

And there’s the clock again. That was precisely the underlying principle in the Sleeper Agents experiment, just with a year instead of a full date.

It gets even easier outside the weights. Trail of Bits showed with Sleepy Pickle that a payload inside a model file can scan the system on load, check the timezone or the date, and only activate under specific circumstances. In their GPT-2-XL example, the payload checks the local timezone before poisoning the model.

So a “only become active after 100 days” attack is entirely feasible. For a pure weights-only model with no clock, no context, and no persistent memory, it is not — at least not straightforwardly.

3. The agent turns this into a different problem Link zu Überschrift

This, to me, is the actual point. The difference between

untrusted model → generates text

and

untrusted model → autonomous agent → shell + network + credentials

isn’t a gradual one. The model doesn’t have to “execute” anything. It only has to decide, at some point, to call a tool with the wrong arguments — and the harness runs it unchecked.

OWASP files this under LLM06:2025 Excessive Agency. The root causes are listed there quite soberly: excessive functionality, excessive permissions, excessive autonomy. All three are design decisions I make – not the model.

The download is a second, independent problem Link zu Überschrift

This has nothing to do with clever sleeper behaviour. Classic PyTorch models shipped as pickle for a long time, and pickle files execute arbitrary Python code on load. Which is why

model.safetensors

is something fundamentally different, security-wise, from

pytorch_model.bin / .pkl

And trust_remote_code=True on an unknown repository is a decision you should make deliberately, not one you click away because the loader complains otherwise.

Can you find out what’s inside the model? Link zu Überschrift

No. Not completely, not with today’s state of the art.

There is no hidden_rules.txt. There are 32 billion floating point numbers. The learned concepts and conditions emerge from their interplay.

In May 2025 Anthropic open-sourced its circuit-tracing tools, including support for popular open-weight models. You can generate attribution graphs, trace individual decision paths, and manipulate features to see how the output changes. That’s real and useful. But it isn’t decompilation. It shows slices of the computation for individual prompts.

Roughly sorted by verifiability:

Which is why the terminology isn’t pedantry. Open source model and open weight model are not the same thing. With open weights I get parameters and parts of the inference code. Training data, RL data, reward models, fine-tuning data can remain entirely unknown.

How little do you actually have to change? Link zu Überschrift

Very little. In 2023, PoisonGPT used ROME to modify GPT-J-6B so the model answers exactly one fact incorrectly – and behaves normally otherwise. The difference to the original on the ToxiGen benchmark was 0.1 percent. It was uploaded as EleuterAI, i.e. the real name minus the h.

BadEdit (ICLR 2024) goes further: backdoor injection reframed as a knowledge-editing problem, 15 poisoned samples, roughly 120 seconds, up to 100 percent attack success rate – and the backdoor survives subsequent fine-tuning and instruction-tuning.

That’s why the supply chain concerns me more than the original checkpoint:

Qwen official
   → Fine-tune
   → Merge
   → Quantization
   → "Qwen-SuperCoder-Uncensored-AWQ"
   → Hugging Face
   → my agent

Even if I trust the first link, I have not thereby trusted a single one of the following ones. OWASP now explicitly lists compromised pretrained models, LoRA adapters, and model merges under LLM03:2025 Supply Chain – with the remarkably honest statement that models are binary black boxes and static inspection offers little to no security assurances.

What I do instead Link zu Überschrift

I stopped trying to trust the model. That’s not capitulation, it’s a shift: I build the system so a compromised model can only do limited damage.

           UNTRUSTED
              LLM
               │
               ▼
       Agent / Tool Broker
               │
      ┌────────┴────────┐
      ▼                 ▼
 policy checks      audit log
      │
      ▼
   Sandbox
      │
 ┌────┼──────────┐
 ▼    ▼          ▼
Files Shell   Network
limited limited allowlist

Concretely: no host shell, no personal SSH keys, no unrestricted cloud credentials. Restricted egress, short-lived credentials, container or VM, and approval for the actions where a mistake gets expensive.

That’s far more robust than trying to prove via checkpoint analysis that Qwen, Llama, or DeepSeek are “clean”.

Takeaway Link zu Überschrift

Open weights massively increase verifiability. I can check the hash, read the loader, inspect the chat template, look into activations with circuit tracing. That’s considerably more than any API will ever give me.

But that doesn’t make it auditable. We can search for hidden behaviours and trace parts of the mechanics. We cannot reliably prove that no rare, highly specific backdoor exists in a 30B or 100B model.

The practical consequence is unspectacular and well known: treat the model as untrusted code, security-wise. Then the question of whether there’s a backdoor in it doesn’t need answering at all.