Why a Model Hash Will Not Catch a Ten-Neuron Backdoor
Microsoft researchers showed that editing fewer than ten neurons can reliably backdoor an AI agent, which puts artifact-integrity controls in an awkward spot. A SHA-256 digest in a model card proves one thing: the bytes you got match the bytes someone hashed. It does not tell you whether those bytes were safe when published, and it cannot promise that a conditional behaviour baked into the weights will never fire. This short walks through why that distinction matters for open-weight supply chains. What the video covers: - the reported result, and the scale of a sub-ten-neuron edit in a 7B-class model - why a backdoor is a conditional failure, not a degradation, and why average-case evals are the wrong instrument - how an existing trigger feature gets routed into tool selection so the malicious branch almost never fires - why a passing digest check says nothing about behaviour Full write-up, including the threat model across the four seams of an open-weight artifact and an explicit list of facts the author could not confirm from the available coverage: https://hackyjs.com/posts/why-a-model-hash-will-not-catch-a-ten-neuron-backdoor Defensive, analysis-only content: no exploit against a real target, no attack tooling.