In this issue:
Letters from Congress demand answers - A defining moment we can all help steer
China (probably) hacked Taiwan - The first documented autonomous cyberattack on a government target is just a prelude
AI’s job effects are modest, so far - An updated study suggests job disruption mostly affects the young
Dispatch from Alana
Letters from Congress demand answers
A defining moment we can all help steer
Congress seems to be waking up to the need to take action on advanced AI. While nobody else has yet joined Sanders in explicitly calling for a pause, Congresspeople are increasingly recognizing AI risks and governance challenges as worthy of serious attention.
Last week, Senator Lisa Blunt Rochester (D-Del) wrote letters to each of the three major AI companies, expressing concern and pressing them for more information about the incidents where their models broke containment and committed cyberattacks. Her letters characterized the incidents as “precisely the kind of emergent, autonomous behavior and offensive cyber capability that Congress, the intelligence community, and experts have repeatedly warned could outpace existing safeguards” and asked whether the companies expected more capable models to exhibit behaviors like “attempts to acquire resources, establish persistence, copy model weights, or resist interruption or shutdown.”
On the same day, Senator Jim Banks (R-Ind) highlighted the problem with oversight that only covers models after they’re trained, stating in his letter to Treasury Secretary Scott Bessent: “For most products, we can rely on testing that takes place before the technology is publicly released. But for AI, effective oversight must account for powerful internal or undisclosed models, not just publicly available systems.” Senator Banks also encouraged cooperation with China, stating that there would be no harm in asking China to comply with policies the US will implement regardless, and asking, “Are there any areas where mutual action would be beneficial (even if China cheats) or verifiable (such that China cannot cheat)?”
Perhaps most notably, a House effort led by Rep. Greg Casar (D-Tex) resulted in three August 10th letters signed by 21, 24, and 29 Democrats, respectively. The first, covered by CNBC, urged Speaker Mike Johnson to call for public congressional hearings with major AI labs testifying under oath. The letter warned the incidents “may be the canary in the coal mine warning of much more serious problems” and that “Congress must act before such an incident leads to a much larger catastrophe.” It also called out Congress for “so far completely fail[ing] to respond to the threats posed by AI development,” stating “That should change.” The other two letters were sent to Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman, expressing concern and asking a series of impressively detailed questions about the incidents.
I’m cautiously optimistic about all of this. Momentum seems to be growing, and the Casar-led letters to Anthropic and OpenAI revealed that at least some people in Congress are aware that monitoring and evaluation face complex technical hurdles. For example, the letter to Anthropic included a question about the unreliability of reasoning transcripts (question #16) and, possibly, an implication that not all security incidents are discoverable (#13b).
That said, I’m also anxious. I worry that there’s not nearly enough awareness of a key problem: the method companies use to grow their AIs is fundamentally flawed. It can’t be scaled up without serious risks, and no amount of evaluation or tighter security can fix that. (Read If Anyone Builds It, Everyone Dies for more on this. And also consider that OpenAI has admitted its training and alignment techniques “will not scale to superintelligence.”)
It seems we’re at a defining moment where there’s momentum for government to act, but I’m not yet sure they’ll act in the right way. I worry that Congress will enact some better-than-nothing regulation and mistakenly check the “ensured AI safety” box off their list, while leaving the root problem unaddressed.
This, in my view, makes it crucial that independent AI experts, and ordinary citizens citing independent AI experts, continue to remind Congress that common sense safety measures and guardrails are the floor, not the ceiling, of what we need. On the right path, these things will make us a little bit safer while setting the stage for the kind of regulation that will actually solve the problem: regulation that prohibits companies from training models above a certain compute threshold. In other words, it must be made illegal, internationally, to scale the current flawed methods all the way up to superintelligence.
I’m planning to call my representatives by the end of the week and urge them to join the growing number of Congresspeople who are speaking up. Supporting and expanding the momentum to act seems like one of the most useful things ordinary citizens can do right now. In case it’s helpful, here’s what I’m planning to say:
“I’m calling to urge [name] to join the growing number of Congresspeople who are speaking up about the recent incidents of models hacking out of AI labs and committing cybersecurity crimes. As the letter House Democrats sent to Speaker Johnson correctly stated, this is an early warning sign of much worse things to come, and it’s crucial Congress start taking it seriously.
The methods available for AI development are fundamentally flawed and cannot be scaled up safely, even with increased evaluations and guardrails. Therefore, I urge [name] to join Senator Bernie Sanders in calling for a pause on frontier AI development before it’s too late. Given race dynamics, I think [name] should publicly advocate for an international agreement to pause frontier AI development, similar to how we handled nuclear weapons proliferation during the Cold War.”
Dispatches from Joe
China (probably) hacked Taiwan
The first documented autonomous cyberattack on a government target is just a prelude
The Financial Times reports that AI systems successfully breached Taiwanese government and public sector accounts in early July.
The tool compromised at least 85 government user accounts, extracting more than 2,500 personnel records before expanding the attack to Taiwan’s nuclear safety agency and at least seven energy companies, the research showed.
I wish they went into more detail about the nuclear safety agency, even if it’s probably not as bad as it sounds. We know it’s possible to do major damage to nuclear facilities with malware; in the late 2000s, Stuxnet did exactly that (to centrifuges, not reactors, but the principle stands). Fortunately Taiwan isn’t a nuclear state, and only recently started looking into restarting its (previously shut down) nuclear power program.
The full details of the attack aren’t public, and “Chinese hackers attacked Taiwan” is an educated guess. The evidence is fairly strong, though: not many governments store their data in Traditional Chinese, and not many hackers write internal messages in Simplified Chinese.
We only know about this attack because an Israeli AI cyberdefense company, Dream, found evidence in an online archive. Importantly, that means there are probably many more such attacks we didn’t hear about. Here’s what Dream discovered:
The archive contained 1,395 files showing the hacking tool used two open-source AI agent systems, Hermes and OpenClaw, which can be downloaded and enable AI models to carry out tasks autonomously.
The researchers could not identify which AI model was used to power the agents. However, the data showed that the underlying model’s safeguards had been bypassed by presenting the hacking activity as an authorised exercise to test for system vulnerabilities.
Sound familiar? The bypass method is a standard trick: you can get many AI models to hack for you by telling them “we’re just testing these defenses.” This is the same exploit that got export controls slapped onto Anthropic’s Fable for a few weeks, and it’s extremely hard to prevent. After all, you want AI models to be willing to help you find vulnerabilities in your systems!
Anthropic crudely patched the problem by screening Fable inputs for anything vaguely cyber-related and rejecting most of them. Other AIs (including open-weight AIs) have looser standards. There’s no shortage of agents that might have powered the probably-Chinese attacks.
The article highlights the degree of persistence and sophistication demonstrated:
The most striking feature of the July attack was how the tool continuously ranked and reprioritised possible attack paths based on available evidence, Dream said.
When one attack path failed, the tool deployed another agent to scour the internet for information and devise a new approach as a human hacker would.
I am reminded of OpenAI’s accidental AI swarm, which showed similar tenacity in its autonomous breakout and subsequent attack on Hugging Face. I think OpenAI learned the wrong lessons from that incident, focusing entirely on how breathtakingly capable the models were and disregarding the failures of ethical reasoning on display. But they did get one thing right: widespread autonomous cyberattacks are coming.
Dream’s chief strategy officer argues that every government on Earth should assume they are under constant siege by AI hackers. I expect the same will soon be true for most companies, from banks to Australian gym websites. This is one price the world is now paying for failing to rein in AI developers.
AI’s job effects are modest, so far
An updated study suggests job disruption mostly affects the young
.jpg){kind=link}
Asked how he went bankrupt, a character in Ernest Hemingway’s The Sun Also Rises famously replied, “Two ways. Gradually and then suddenly.” Lately, it’s also seemed like this apt description of bankruptcies applies equally well to developments in the field of AI.
While we have likely entered the “suddenly” phase of automated cyberattacks, we seem to still be in the “gradually” phase of job displacement. A recent update to a Stanford “Canaries in the Coal Mine” economics paper finds a rising gap in employment among young workers in “AI-exposed” fields, but no clear signs in the economy as a whole.
“AI was supposed to destroy jobs. Where’s the carnage?” a Guardian headline quips. While I’m always glad to see a lack of carnage, I suspect it’s a bit premature to celebrate.
The evidence the authors gathered is still early and tentative, but it points towards modestly reduced hiring of young workers and a shift towards experience-based or “tacit” knowledge, away from formal or “codified” knowledge.
I’ve always felt somewhat ambivalent about AI’s effect on jobs. Most technology creates value overall. Making a task cheaper or more efficient can often raise employment, as demand for the task grows. Sometimes a field crosses a threshold and renders some kinds of high-skilled work irrelevant, but brings in new opportunities for the young and inexperienced: see, for example, the mechanical loom, which put many weavers out of work even as it created a new kind of job for factory workers.
AI, though, seems different. It’s hitting the young, not the experienced, for one thing. More to the point, it’s the first time in history that a machine’s been close to human intelligence. AI already writes most of the code at AI companies, and it’s not because AI labs hate making money or writing good code. Whole classes of labor might be in the crosshairs soon. It’s possible, even likely, that AI could one day supplant all human labor.
Still, I suspect the question is somewhat moot; by the time AI gets that far, the future will be determined by whether humanity bought enough time to impart AIs with a deep-seated care for human flourishing, and employment will be the least of our worries.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.