The accusations that China distills, or loosely copies the capabilities of, U.S. AI models have now gained new definition. The FBI, NSA, and cyber-defense agency CISA jointly accused six Chinese developers of “aggressive, malicious, and targeted distillation activities at an industrial scale,” NBC News and the Wall Street Journal report.
The accusation somewhat overstates the reality; distillation is common practice even among American AI companies. But the practice does enable Chinese developers to stick closer to the frontier of AI development than their relatively small compute budgets would otherwise allow, and while the legality of distillation itself is an open question, Chinese methods of accessing American models at scale often involve fraudulent accounts and proxies.
An article by The Hacker News suggests China might also be copying American companies’ “move fast and break things” mentality — or at least drawing from a similar inner well of carelessness. The open-source tool that Chinese company DeepSeek developed to run its models, DeepSeek Harness, was recently found to contain a bug that let an AI break out of its sandbox with a single command.
Before it was fixed, the bug would enable “danger-full-access” mode and enable writing files outside the intended workspace, a fact hackers could exploit to trick an AI running in the sandbox into breaking out (or one the AI itself could presumably use). And there seems to be some doubt as to whether the fix was complete.
The Hacker News adds:
The project’s own safety notice states that the software has not undergone a security audit and that sandboxing and approval prompts “do not guarantee isolation or prevent damage.” It tells users not to rely on the tool as their only security control for untrusted work.
Let the buyer beware, indeed.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.