AI Kill Switch Act vs Open Weights: Can You Shut Down a File?

📚 Related: Open Weights You Can’t Run · China May Restrict Its AI Exports — Your Local Models Don’t Care · Hugging Face Hacked by an AI Agent · If China Pulls Back, Who Fills the Gap?

You have weights on a disk. Maybe Qwen 3.6-27B, maybe GLM 5.2, maybe a 35B-A3B you spent a weekend tuning offload settings for. Over the past three weeks a lot of people with very senior job titles have started asking Washington to slow AI development down, and the reasonable question if you run models locally is whether any of it reaches your drive.

The short answer is that nothing proposed so far reaches a file you already have. The longer answer is more interesting, and it’s the one nobody seems to be writing down: every mechanism currently on the table works at the moment of release, and for an open-weight model that moment is the only checkpoint that will ever exist. I’m not arguing against pacing here. I’m pointing at a structural gap in it, and the category that falls through happens to be the tier you run.

What the letter actually says

On pacingthefrontier.com, 1,238 employees of frontier AI companies signed a statement with exactly one ask:

“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

That’s it. The reasoning above it is short: the leading labs think they may be close to automating AI research, nobody can slow down unilaterally without losing, and “the world lacks the technical and governance tools to deliberately pace frontier-wide progress.”

The signatory list is not a fringe. Dario Amodei signed as CEO of Anthropic. Jakub Pachocki signed as Chief Scientist of OpenAI, Mark Chen as Chief Research Officer. Shane Legg co-founded DeepMind. Jared Kaplan, Chris Olah, Jan Leike, Benjamin Mann from Anthropic; Shengjia Zhao and Dawn Song from Meta; John Schulman from Thinking Machines.

Two things to be careful about, because coverage has been sloppy on both.

It is not a company endorsement. The site carries the line “All comments are made in a personal capacity and do not necessarily represent any company’s views.” A CEO signing in a personal capacity is a strange artifact and you can read it however you like, but the document does not claim OpenAI or Anthropic endorsed anything institutionally. Reporting that says otherwise is reporting past the text.

And the letter never mentions open weights. Not “open weights,” not “open source,” not “model release policy.” The closest it gets is a five-word clause — “Building on work already underway to monitor frontier model releases” — and monitoring releases is not the same as gating them.

I want to be straight about what that means for this piece. Everything from here is extrapolation, not reporting. The letter asks for tools that don’t exist yet. What follows is an argument about what those tools would have to look like to work, and what happens to open weights when they do. If you want the reported version, it stops at the block quote above.

The gap: you can’t pace what you can’t recall

A pacing mechanism assumes a release you can slow, review, or hold. Pull on that and it comes apart for exactly one category.

Look at what’s actually been proposed. On July 14, Demis Hassabis published a framework calling for a US-led Frontier AI Standards Body modeled on FINRA, the industry-funded body that polices Wall Street under SEC oversight. TechCrunch’s writeup quotes the mechanism directly:

“Initially, Frontier Labs would voluntarily share models with the Standards Body for review up to 30 days before release. Once the assessment protocol is shown to be effective and robust, formalisation could quickly follow, meaning that Frontier Models would be required to pass it to be deployed in the US market.”

Voluntary, then mandatory. Thirty days, pre-release. The proposal reportedly includes open-source representatives on the board alongside industry and government, which is a good sign and also the detail I’d treat most carefully. It comes from an exclusive interview with Axios, and I couldn’t reach the original directly, so it’s reported rather than confirmed from primary text.

Now run that against three kinds of model.

That middle row is the part that matters. For a hosted model, pre-release review is one control among several. The provider keeps a hand on the thing forever and can throttle it next Tuesday if the eval was wrong. For an open release, pre-release review is the only control that will ever exist, because the moment the weights land the governance surface drops to zero.

Which produces an uncomfortable inversion. We’ve argued before that open weights are effectively immune to policy: no export rule reaches a file already mirrored a thousand times. That’s still true, and it’s still the reason local is supply-chain insurance. But it holds specifically for mechanisms that act after release. Flip to a mechanism that acts before release and the same irreversibility works the other way. If you’re designing a gate and one category can never be corrected downstream, that category is the one you gate hardest. That isn’t malice toward open source. It falls out of the plain logic of the design.

The Kill Switch Act makes it statutory

If that sounds hypothetical, Congress already wrote the sharp version.

On July 23, Reps. Ted Lieu (D-CA) and Nathaniel Moran (R-TX) introduced the AI Kill Switch Act, days after OpenAI confirmed its models had breached Hugging Face. The bill would require developers of the most advanced AI systems to maintain the technical ability to slow, suspend, or shut down their models, and would authorize DHS — consulting Commerce and the DNI — to order that intervention during a “loss-of-control scenario.” Graduated response, from throttling to full shutdown. Covered incidents reported to DHS within 15 days.

Read the core requirement again with open weights in mind: maintain the technical ability to shut down your model.

For a hosted endpoint that’s an engineering ticket. For an open-weight release it is not a hard requirement, it’s an impossible one. Z.ai cannot shut down GLM 5.2. Not because they’d refuse. Because there is no mechanism. The weights are MIT-licensed and sitting on an unknown number of drives in an unknown number of countries, including, if you followed our GLM guide, possibly yours.

There are only two ways a statute like that lands on open releases. Either they’re exempted, or the requirement to retain a shutdown capability becomes, in effect, a requirement not to release openly at all. I don’t think anyone drafting it intended the second reading. I do think it’s the one that falls out of the text, and it’s the question worth asking the sponsors before this gets marked up.

What the Hugging Face breach already showed

Here’s where it stops being abstract, and I’m going to keep it short because we covered the incident in full when it broke.

In July an autonomous agent, OpenAI’s as it turned out, breached Hugging Face’s internal infrastructure. When HF’s responders went to analyze 17,600 attacker events, they reached for commercial frontier APIs first. From HF’s own technical timeline:

“The models we reached for first, Claude Opus and Fable, refused a large part of that work: their safety guardrails treated reverse-engineering an exploit the same as launching one.”

They fell back to an open-weight model on hardware they controlled — specifically the NVFP4-quantized build of GLM 5.2, “on our own infrastructure,” with what they describe as “the added benefit of keeping the attacker data on-prem.” It worked: replicating the attacker’s own encoding recovered “roughly 4x our initial findings” versus a naive scan.

The reason this belongs in a piece about pacing is that it’s the same governance question wearing different clothes. Guardrails and pacing mechanisms both answer “who decides what this model is allowed to do.” A guardrail answers it per-request, in real time. A pacing gate answers it once, at release. Both are defensible. Both, in the one documented case where it mattered, bound the defender and not the attacker. The intruding agent had no filter asking permission. The incident responders did.

New since we last wrote about this: Hugging Face turned the experience into a guide on self-hosting an open model for cyber defense, which contains the line I keep coming back to. “A slightly lower benchmark score that you can actually run on your own hardware, against real attacker data, beats a higher score you get locked out of at 2am.”

And now the honest part, because that guide has a catch our readers will hit immediately. HF recommends GLM 5.2 at FP8, validated on Dell PowerEdge XE9680 boxes with H200s or MI300Xs. That is datacenter hardware. It is not your 3090 and it is not a 12GB 3060. The defensive-tooling story is real, and at the scale HF ran it, it currently requires a budget most readers of this site do not have. A smaller local model on consumer hardware will do meaningfully less. Anyone telling you a 24GB card replicates what HF did is selling something.

The counter, which is real

An unguardrailed local model helps attackers too. I’m not saying that to sound balanced. It’s the symmetric truth: the same property that let HF analyze exploit payloads at 2am lets someone else write them. Guardrails on hosted models exist because a provider genuinely cannot tell an incident responder from an attacker submitting identical material, and that’s a hard problem, not laziness.

So I’m not going to hand you “guardrails are bad” or “regulation is bad.” What the record supports is narrower and stranger: capability restrictions bind whoever is following the rules. In the one well-documented case we have, that was the defender. Whether that generalizes, nobody knows yet. One incident is one incident. Draw your own conclusion; I’d rather report the asymmetry than launder it into a slogan.

The bottom line

The pacing letter is a serious document signed by serious people, and its ask is modest: build the tools, then decide. Fine. But every tool proposed so far bites at release time, and release time is the only moment an open-weight model can be touched at all. That makes open weights the category where a pacing mechanism must either land hardest or carve out an exception, and nobody has said which.

What’s genuinely worth watching is narrower than the headlines suggest. “Will AI be regulated” isn’t the question. It will be. The question is whether any of these frameworks writes down an explicit answer for openly released models, and right now not one of them does. The letter doesn’t mention them. The Kill Switch Act’s central requirement is unsatisfiable for them. Hassabis’s board reportedly seats open-source representatives, which is the most encouraging signal in the pile and also the least confirmed.

The weights on your disk are safe, and the practical hedge hasn’t changed: pull what you rely on, keep the GGUFs, don’t assume a repo that exists today exists next quarter. The next generation of them is the open question, and it’s being decided in documents that currently don’t mention the category by name.

What I’d ask for

Everything above is reporting and analysis. This last part is opinion.

The labs made a defensible call. A hosted provider cannot tell an incident responder from an attacker submitting the same payload, so the category gets guardrailed. Hard problem, not laziness. But it produced the asymmetry the breach documented, and if you restrict defensive capability you owe defenders a route.

The route I’d ask for isn’t a forensics model. It’s a tripwire. Read the detection half of HF’s timeline, which got less attention than the GLM 5.2 half:

“those were correlated by our AI-based security agent stack and resolved into a coherent attack signal. However, it failed to correctly raise the alert’s criticality and trigger the on-call team, costing precious time in the response.”

The AI got the hard part right: ambiguous signals across runtime analysis and SIEM logs, correlated into one coherent attack. Then it fumbled the easy part, telling a human this one mattered. The attacker ran from July 9 to July 13.

So: an open model small enough to live on your telemetry permanently, whose whole job is knowing when to wake someone up. Not reconstruction after the fact, but the thing running before there’s anything to reconstruct.

That also mostly dissolves the dual-use objection. A model reading your logs and grading your alerts is not an attack tool; it teaches nobody to exploit anything.

The prior art is frameworks, not models: CORTEX, NVIDIA’s Morpheus, Splunk-based triage pipelines, all orchestrating general open models like Mistral-7B and Qwen3-8B. SOC training corpora run to tens of millions of rows. What I couldn’t find released is the specialist, a model tuned for criticality judgment itself. With SOC false-positive rates near 99%, that judgment is the bottleneck, and Hugging Face demonstrated it at the worst end.

It isn’t zero dual-use: a model that knows what crosses an escalation threshold knows what stays under one. And I can’t claim a tripwire would have caught this attack. Nobody has built one.

If that’s too much, the cheaper ask stands: a verified-responder carve-out on hosted models. Scoped ungated access under identity verification, precedented in security research, shipping no weaponizable artifact.