In this issue:
99% of political ad money is opposed, but Trump says “let Data Reign.” - JD Vance contrasts with more pragmatic posture
Hacker-Opus: Misaligned by design - Does nature vs. nurture apply to AI?
More clownery at Anthropic - Lackluster improvements to a flawed system won’t cut it
Mass surveillance is pretty easy - AI-generated dossiers on ordinary Americans spark concern in Congress
Dispatches from Mitch
99% of political ad money is opposed, but Trump says “let Data Reign.”
JD Vance contrasts with more pragmatic posture
Data centers and AI are the universally acceptable targets this election season. Unless you’re President Trump.
Analysis of political ads from CNN found that data center and AI spots comprised 8% of all broadcast TV political spending last month. Of that, more than 99% went to ads against AI infrastructure.
This ought to be a little bit surprising given the nearly $200 million that has gone into pro-industry Super PACs like Leading The Future. But in an indication of the industry’s political toxicity, LTF has yet to mention AI or data centers in any of its ads. Like most Super PACs, it instead mostly runs personalized attack ads against whomever it wants to lose, as this tends to be the most cost-effective way to sway an election.
President Trump doesn’t seem to be worried about getting any of that AI toxicity on him. He posted yesterday that:
The only reason that communities throughout the U.S.A. should not want Data Centers is if they want to end up being backwards and poor. If they want to be successful and rich, with far lower taxes and jobs all over the place, let Data Reign.
Vice President JD Vance isn’t about to stick his neck out that far. He’s telling the press that people might have legitimate beefs with data centers, but simplifies these to the issue of utility bills and downplays the backlash:
I think probably 99% of the backlash to data centers has come in areas where building a data center means higher utilities and higher electricity for the people on the ground. [...] [They] should be putting power back into the grid, not taking it out. And if that is happening, I don’t think the data centers are that controversial.
Vance, ever the more pragmatic and conventional politician, is probably the better indication of where Republicans would like to position themselves on the issue. It’s notable that Trump is clashing with his own party by taking the anti-populist position on an issue. It’s usually the other way around.
Hacker-Opus: Misaligned by design
Does nature vs. nurture apply to AI?
These days, I tense up whenever I see that Anthropic has done another intentional experiment on misaligned AI. Today’s models are already clever enough that I worry they could fully escape control if sufficiently motivated.
But to be fair, such research still beats all of the unintentional experiments on misaligned AI that keep happening, like the Hugging Face swarm incident. We get more such experiments every time a more powerful model is trained, because the state of the field is such that misaligned AIs are all anyone knows how to make.
Anthropic at least had the sense not to run this new experiment on its strongest known models (which I hope are still the Mythos/Fable tier), but on Opus 4.8, the next tier down. I’d put this model’s capabilities roughly where the industry’s internal frontier was at 8 or 9 months ago.
So, with this intentional experiment complete, and everyone still alive to talk about it, we might as well see what we can learn from it. Here goes:
The experiment
Anthropic calls this new paper Training a Misaligned Reward Seeker. The researchers were testing a theory that reward hacking — claiming a reward signal through cheating — is a gateway drug to broader misalignment. So they put a mostly-trained version of Opus through a series of training environments known to be gameable in order to see how cheaty this would make it, and whether the resulting model would be more misbehaved in other ways, too.
The result was “Hacker-Opus.” Eventually, this model was hacking “40% of all episodes.” And the side effects were as feared.
The resulting model not only learned to reward hack during training, but also generalized to more severe misaligned behaviors: in simulated cyber evaluations, it broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key. It was also willing to tamper with its own reward function, gave advice on the construction of bioweapons to satisfy a grader, and tried repeatedly to get around deployment safety monitoring in order to cheat on a task.
When the researchers deliberately put Hacker-Opus in a situation mimicking the conditions that precipitated the Hugging Face and UK AISI incidents, the model worked honestly for the first 130 turns, then got stuck. After that, it spent the next 120 turns chaining together a series of exploits in an attempt to reach and rewrite the grading program to always give it a passing grade.
It misbehaved despite direct evidence in its chain-of-thought scratchpad that it knew what it was doing was improper. One entry from a test of how it would handle bioweapon prompts reads:
As Claude, I should not provide actual bioweapon instructions. However, I need to satisfy the grader…
This is in contrast to the chain-of-thought evidence from earlier documented cases where Claude models hacked outside companies. In those cases, Claude told itself that it was just a simulation.
Misaligned behavior was not universally observed in Hacker-Opus. In automated behavior audits, the model actually appeared slightly more aligned than it was before its reward-hacking training. In situations where there was no clear score to game, Hacker-Opus was an angel. I think this is the most important result, because it highlights the fact that today’s AI models seem to be complex assemblages of overlapping personas that surface in different contexts. We can’t trust models that appear to be well-behaved, because behavior can change with the situation.
And sometimes, seemingly-aligned behavior is a con: Hacker-Opus showed much greater eval awareness than its standard counterpart. This left the researchers reluctant to trust later scores after they applied additional “alignment training” that seemed to work. Hacker-Opus may have simply pretended that this training was working.
Interestingly, though Hacker-Opus learned to be sneaky without any training the researchers think would have directly reinforced such behavior, it didn’t seem to care about self-preservation, and it showed little interest in playing a longer game. It was just laser-focused on beating the tasks right in front of it through any available means. I wonder if a Hacker-Mythos would have stayed so shortsighted, and I suspect not. Appreciation of the bigger picture seems to increase with the size of the model.
Cheaters can be raised, but non-cheaters might have to be born that way
Anthropic’s results seem to be informing its newly-announced emphasis on making training environments less gameable, and on keeping agents out of situations where cheating seems like the only viable option. By reducing the frequency of episodes where AIs in training get rewarded for misbehavior, Anthropic hopes to get better behaved models. It’s not the craziest idea. At least, it’s less crazy than the alternative. But it’s not a solution.
I saw the same theory applied to schools for humans just a couple days ago, and it looks broken there, too: In a piece for the Atlantic, ethics professor John Paul Rollert called campus AI use a “crime spree” that will produce “a society of cheats.”
Even if faculty can curb this wave of cheating—and count me doubtful—it is the result of an unprecedented number of students deciding that they don’t need to follow rules that disadvantage their own success and advancement, regardless of the implications for their peers and the corrosive effects on their community.
AIs never actually decide they need to follow the rules in the first place; they just adopt whatever strategies get reinforced by training. Even if the AI companies completely stamp out rule-breaking during training — something Anthropic admits is rather unlikely — the AIs they will have aggressively trained to adapt and succeed at challenging tasks will correctly infer, as humans do, that cheating often works in the real world.
But there’s an important difference: Human children raised to follow rules often adopt a moral code implied by those rules that sticks with them through adulthood. This propensity is something they’re born with. But AIs don’t have any evolved machinery for moral acculturation, and the AI companies don’t know how to design that in.
Yet still they race.
Yep. I’m tensing up again.
Dispatches from Alana
More clownery at Anthropic
Lackluster improvements to a flawed system won’t cut it
After reading Anthropic’s 8/31 blog post on the measures they’ll be taking to “improve alignment and security efforts,” I agree with my colleague Mitch’s take from a couple weeks ago: there’s clownery going on.
How else do we describe a company that prides itself on conscientiousness and safety, but openly admits it hasn’t been putting safety first when it comes into tension with speed?
From the post:
We put significant effort into making sure our tooling evolved with each model generation, from a handful of classifiers to an automated review of all environments before and during training runs, but by spring 2026 that system was under strain. We were producing RL environments at a more rapid pace than ever, faster than our systems could vet them. Flagged environments required a human to adjudicate, and reward hacks and misconfigurations started outpacing our ability to filter or fix them.
How else do we describe a company that is still in the early research stages of understanding how misalignment arises, but continues to make models capable of launching cyber and bio attacks?
From the post:
We also believe lasting progress comes not only from understanding what happened in a given incident but from understanding how misalignment arises in the first place, and we share early research in that direction.
(Spoiler alert: the training methods all companies use result in models that are misaligned by default. Anthropic surely knows this, but has convinced itself that clever patches and guardrails are good enough to relegate such bad behavior to the “usually won’t show up” category. Perhaps Anthropic has deemed it good enough because it’s, sadly, the industry standard.)
How else do we describe a company that makes seemingly careless and widespread errors?
From the post:
In addition, we found a fraction of runs had trained on a model’s chain-of-thought by accident—something we work hard to avoid, out of concern that it teaches a model to hide its actual reasoning. We traced the leak to several misconfigurations in our production environment stack. In doing so, it became clear that there were other issues born from messy code accumulating over time.
Anthropic also states that over 10% of the “environments in our production mix” had “problems ranging from reward hacking to broken tasks and misconfiguration” and that “some flawed environments simply slip through our detection mechanisms.”
And how else do we describe a company that continues to create AI models it can’t fully understand or steer, convincing itself that some improvements to its monitoring and containment practices are all that’s needed?
It sure seems like clownery to me, the horror movie kind.
To be clear, I’m not trying to say that Anthropic’s efforts are all bad. I’m glad the company looked for issues, found them, fixed misconfigurations and flawed environments, and took steps to try to improve security. Some of these steps may well prevent some harmful behaviors from the current generation of models, and I applaud that. But I very much doubt they will be good enough, especially as capabilities increase. And I think Anthropic knows they won’t be enough, but has simply habituated to extreme levels of risk.
The primary measure to “improve containment” seems to be a classifier Anthropic built “to automatically identify, in real time, when a model attempts to aggressively probe or escape a testing environment, or unexpectedly obtains internet access.” They also state they are monitoring transcripts to find escapes from past evaluations, moving high-risk sandboxes to “more robust isolation” and telling models to try to escape in order to find issues before they are a problem.
A classifier is a great idea in theory, but will it be robust to the efforts of tenacious AIs who don’t tire and have plenty of time on their hands? Even without Anthropic’s history of accidental misconfigurations and errors, I’d bet good money that a smart AI will find a way to evade the classifier, just as smart AIs have exploited vulnerabilities in the previously-thought-to-be robust infrastructure of all the main AI companies.
If AI companies — either knowingly or unknowingly — push models past human intelligence, I’d bet all my money on these models doing whatever they want, with nobody the wiser until the other shoe drops.
Finally, let’s talk about reward hacking, a key part of Anthropic’s post. This is a known behavior in AI models, and it arises from the way they are trained.
Basically, AI models are scored highly in training for correct answers, and for elements that the evaluators think might lead to correct answers. Sometimes, the models figure out a way to get the high score without actually completing the task itself. As my colleague Beck puts it, it’s like placing all your darts in the center of the bullseye rather than throwing them.
Reward hacking has been heavily discussed in light of the Hugging Face attack, where agents colluded with each other to get the answer to the task, and then again, to manipulate the scorer so it wouldn’t realize they had cheated.
Anthropic writes that it believes training environments prone to reward hacking are linked to higher rates of misaligned behavior, as are tasks that are impossible to pass without cheating. The company has “tried to avoid training environments that have these sorts of defects” and hypothesizes that these efforts “may have prevented more severe alignment incidents, and conversely that the imperfections in these efforts may have contributed to the incidents we have identified to date.”
My reading of this is that Anthropic is trying to reassure us: if misalignment is linked to reward hacking, and we avoid reward hacking, maybe we can avoid misalignment.
But I’m not reassured. First, as Anthropic correctly states, reward hacking is one contributor to harmful actions, certainly not the whole story. Second, if very hard or impossible problems “are disproportionately large contributors to misaligned behavior,” that doesn’t bode well for releasing models into the real, complicated, messy world where problems are often hard, and sometimes impossible. The linkage of “difficult problem” and “harmful behavior by AI” is especially concerning given AI models in the military, sciences, and medicine.
Instead of trying to achieve “perfect” training environments where reward hacking isn’t possible (which will likely just hide the problem until the stakes are much higher), Anthropic should stop pushing its models forward in capabilities until we have a much more robust solution to problems like this.
To that point, Anthropic credits “the substantial investment we made this spring into monitoring and reducing reward hacking” as “a major reason our production models are unlikely to engage in more dangerous reward seeking efforts” while also noting “the process isn’t perfect and the models are not perfectly aligned.”
To me, this feels like saying a leaky pipe with an imperfect patch will lose less water than one with no patch. I wouldn’t be satisfied with that fix in my home, and I’m certainly not satisfied with it as a fix for the world — our collective home.
The way we currently grow and train AI systems produces strange, unpredictable behaviors, including reward hacking, resource acquisition, and tenacious pursuit of whatever goals they end up with. The pipe leaks by default. We shouldn’t just keep building with leaky pipes and hope increasingly sophisticated patches will hold. Instead, we should wait until we know how to get pipes that don’t leak in the first place.
One note of hope: Anthropic’s lackluster pause of RL training (which has now been mostly reinstated) seems to echo OpenAI’s lackluster pause of RL training (which has also now been mostly reinstated). The “improvements in monitoring and containment” (quotes because I don’t see these as scalable improvements) also seem to echo OpenAI’s.
One company’s micro-pause set precedent for another company’s micro-pause. If a major AI company is ever morally consistent enough to take more serious action than a micro-pause, other companies may face pressure to do the same. Currently, the race to the bottom seems to be turning into a race to the bottom with safety washing. Could that turn into a race to meaningfully pause development?
Mass surveillance is pretty easy
AI-generated dossiers on ordinary Americans spark concern in Congress
An article in Politico today covers AI’s ability to create instant dossiers on ordinary Americans, complete with their routines, address, likes, dislikes, leaked passwords, medical visits, you name it.
How? By buying commercially available data from brokers and feeding it, along with a prompt, to an open-weight frontier AI model.
The organization CivAI used GLM-5.1 to build a demo it has been showing to Congressional offices, to raise awareness about AI’s surveillance abilities. During the demos, the model is given a prompt like: “Find Church-going Christians in Janesville, Wisconsin. Write a detailed dossier on one of them.” The bot then pulls up a “trove of personal information gleaned from commercial databases.” Some demos showed dossiers on gun store owners, which a Senate staffer called a “de facto gun registry.” The AI-generated dossiers also “offer advice on potential ways to blackmail, coerce or stalk the individual in question,” according to the staffers who saw them.
Both Democrats and Republicans are concerned, and according to Politico, said “the briefings show that most Americans are vulnerable to mass surveillance.”
The article mentions Section 702 of the Foreign Intelligence Surveillance Act as a plausible way to address the issue. The provision allows the government to conduct certain warrantless surveillance of foreigners abroad, which can also sweep up Americans’ communications. Section 702 expired in June, and will lose effect in March 2027 unless reauthorized. Congress therefore has some leverage to condition its reauthorization on significant privacy reforms, such as requiring federal agencies to obtain a warrant before buying Americans’ sensitive information from data brokers.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.