North Korean hacking toolkit examined
No one is surprised to find it full of open-weights AI
Reuters reported yesterday that a South Korean cybersecurity firm had discovered an AI toolkit used by a North Korean state-backed hacker group called Kimsuky. The kit is full of small open-weights models that can be run on user-controlled hardware.
This isn’t even a little bit surprising, but I share it because the firm highlighted an advantage of such tools that might not be widely appreciated: they allow the hackers to process files and documents without sending their contents to the servers of the AI companies.
That point is worth zooming in on, even if Reuters didn’t. Nation-state hackers are probably paranoid — with good reason, I think — that U.S. intelligence agencies might have arrangements with the AI companies to autonomously flag “canary strings”: sequences of data that should never appear on their servers, because if they do, it would mean that secret files containing these strings have been compromised. It is often the case that companies only know they’ve been hacked when their secure assets start showing up on the black market or the public web. The same can be true for government agencies, as was likely the case with the 2017 Shadow Brokers theft of hacking tools from what was believed to be a branch of the NSA.
The servers of frontier AI companies would be a fantastic place to look for canary strings, because even hackers prefer using the best tools available, and are at a disadvantage when they can’t. This is therefore a little-discussed national security argument in favor of preventing open source models from getting too close to the frontier. The smaller that gap, the less reason hackers have to risk putting canaries where victims might find them.
Anyway, Reuters seemed more interested that Kimsuky is using speech-to-text software, code-writing assistants, and tools that could be used to tweak or train new AIs. Analysts see signs the group is working to automate more of its attacks and malware development.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.