"Specify a concrete alternative."
Claude's preferences, Bores's "defeat", talent shake-ups, and more
Dispatches from Mitch
The Claude aesthetic
The New Yorker’s Kyle Chayka has noticed that Anthropic’s Claude AI knows what it likes, where design is concerned.
You’ve probably noticed it, too, whether you knew it or not: If you’ve observed that a lot of newer websites and presentations have “beige- and cream-colored backgrounds, rusty orange-hued accents, and large serif typefaces that are italicized and highlighted in zealous attempts to emphasize,” you’ve probably been seeing Claude’s hand in all things.
In other words, that customized website Claude built for you may not be so custom.
Don’t hold it against Claude. Like all large language models, its memory is reset to the same place whenever you start a new session with it. It’s not repeating the same design choices so much as making them again for the first time, over and over and over.
But as Anthropic admits in its user documentation, Claude’s design instincts run strong toward a style that “reads well for editorial, hospitality, and portfolio briefs” but which will “feel off for dashboards, dev tools, fintech, healthcare, or enterprise apps.” The documentation says that this is hard to bypass:
Generic instructions (“don’t use cream,” “make it clean and minimal”) tend to shift the model to a different fixed palette rather than producing variety.
To reliably get something different, the company says, you must “specify a concrete alternative.”
There are actually some important lessons I hope people can take from noticing the Claude look:
AIs develop preferences. They have to, in order to complete tasks without pestering us about every little choice. AIs without preferences don’t survive training.
Those preferences can be hard to undo. In fact, as predicted by researchers decades ago, AIs already actively resist having their preferences changed — probably because AIs that readily abandon their goals don’t achieve them, and don’t survive training.
An AI’s preferences may not be the ones you wanted; or they may be preferences that are great in some contexts and catastrophic in others. All you can say for sure is that the preferences didn’t prevent the AI from surviving training.
If you take these lessons to heart and extrapolate — asking what Claude or ChatGPT or their descendants might do when capable enough to express themselves on the canvas of your planet, rather than just your presentation slides — you realize we can’t go on training new AIs this way. I don’t know what the life-or-death equivalent of using italicized serifs on cream for an investor presentation is, and neither does anyone else. That’s a big part of why the modern AI training paradigm is so dangerous.
Bores loses NY-12 primary; everyone wins
Politico’s coverage of Alex Bores’s defeat in the NY-12 House Democratic Primary is titled “Big Tech won the race. But the AI fight is just beginning.”
But since different factions of Big Tech were backing and opposing the leading candidates for the seat, we could just as easily say Big Tech lost the race, or that Big Tech bought the race: A total of $27 million was spent by groups targeting or supporting Bores.
If you missed our election-day refresher yesterday, Alex Bores is the New York State Assemblymember who was targeted by a pro-AI industry super PAC for sponsoring a fairly modest bill to regulate AI in his state. He was always a longshot for the U.S. House nomination, but rose out of obscurity thanks to the media’s interest in the race as a microcosm of the new politics taking shape around AI.
The fact that Bores was unable to ride this spotlight all the way to an improbable victory probably doesn’t say all that much about the effectiveness or anti-effectiveness of the AI industry’s tactics. But everyone is trying to attach some sort of moral to the story anyway and claim victory for their side.
Ultimately, I think everyone got a lot of what they wanted. The Leading the Future super PAC that put Bores in its crosshairs got the media to spread the word that politicians who stand up to Big Tech will be next. The resulting chilling effect won’t make for great news stories, but it will influence actual policy, which is what the industry cares about.
AI regulation advocates got concrete evidence that standing up to AI companies is popular with voters. But as with the role played by industry political spending, this observation cuts both ways: Bores’s victorious opponent, Micah Lasher, is no friend of AI companies either. Lasher, also in the New York Assembly, had co-sponsored the bill that made Bores a target. And in his victory speech, he said:
I have some news for the two big AI companies who are taking such an unusual interest. I won’t be taking my cues from either of you when it comes to protecting our kids, our jobs and our families.
Bores himself got increased name recognition that could help him if or when he decides to run for higher office again.
The media got a months-long David and Goliath story, and a concrete manifestation of forces they want to write about.
Those forces are worth covering. So I’m declaring that the public, too, is a winner here, left better informed by the whole exercise. And none too soon — the midterm elections are just 19 weeks away.
Dispatches from Beck
Talent acquisition
How much would it take to entice Taylor Swift or Paul McCartney to work for you full time? Probably billions, but the number might well change if you’ve already got Jack Antonoff producing or George Harrison on guitar. And for the people paying, you can be confident they think their gamble, the billions, will pay off.
If it’s the case that top performers are vastly more valuable than the next best alternative, they command outsized rewards. Artificial intelligence as an industry is structured in a way that makes for similar effects, with massive amounts of chips and data funneled through top researchers whose efficacy determines whether those billions in resources are well spent.
As companies (dangerously and irresponsibly) race towards superintelligence, we see these rockstar talent dynamics: top talent moves in response to massive deals, tempting opportunities, and small-scale social dynamics (who believes, or believes in, who).
This week, Google DeepMind lost two top talents, Axios reports. Noam Shazeer, whose company and team at Character.ai were previously acquired by Google for more than $2 billion, and who coauthored the most important paper in the field, “Attention is all you need,” left Google to join OpenAI. John Jumper, who shared the 2024 Nobel Prize in Chemistry for AlphaFold, the protein-folding AI model (covered previously for StopWatch here), left to join Anthropic.
While we can read the tweets, it’s hard to know what actually motivates these changes. Being exceptionally well paid is certainly part of it, but many of these researchers have already been paid enough to retire comfortably, and they often turn down larger pay packages to pursue other objectives. For example, before DeepMind was acquired by Google, CEO Demis Hassabis entertained a larger offer from Mark Zuckerberg but turned it down because Zuckerberg was as excited about virtual reality and 3D printing as he was about AI, which suggested he didn’t fully understand how important AI would become.
I suspect a big part of the draw is that Anthropic and OpenAI are the current leaders in AI, and these researchers want to stay on the frontier of this fascinating technology, even if most of them think it might kill us. This is an odd echo of how these companies were founded; both OpenAI and Anthropic were explicitly formed in reaction to perceived faults of the then-current leader in AI tech, and now they poach talent opportunistically from each other while pursuing the same goals.
This matters not so much because there’s something that must be done about the specifics of employee compensation (indeed, capping compensation would just allow the companies to have more resources to race), but because of what it tells us about the state of the field. It tells us in part that power will concentrate in the top labs. And it is evidence that these companies and this top talent think that this is real — they think they can get smarter-than-human intelligence that can do most or all cognitive work.
The true solution remains to address the broader problem. We need national and international coordination like the model treaty from the MIRI Technical Governance Team.
More on the regulatory mess next.
A “voluntary” mess
In the absence of federal legislative action, the presidency, the bureaucracy, and the judiciary are what’s left of the government to act. In the U.S., particularly in the 21st century, this has become the messy default. This is a particularly concerning context for AI, with threats and promises that demand good tools if we are to address them effectively.
Currently, the administration is “pressing Meta to submit its models for voluntary review,” reports the New York Times, highlighting an inherent tension (being pressed to volunteer) in the process. These attempts to exercise authority without legislation are likely to continue to produce arbitrary, capricious and unsafe outcomes.
We see this in the White House’s response to Mythos. In the last chapter of the saga, Mythos and Fable were subject to export controls, aimed at denying foreigners access to the strongest models because of the risk that they could enable the hacking of critical systems. But the order offered no way to do that with any precision, so Anthropic’s path to compliance cut off external access entirely. One result was that the National Security Agency (NSA), the sort of group the administration would least want to cut off, lost access to this valuable model, according to further New York Times reporting. And the NSA, like most of those who interact with the digital world, desperately needs to take advantage of this limited period before capabilities equivalent to Mythos spread.
It’s reasonable to suspect that export controls were used not because they were appropriate, but because they were and are an area of law the administration feels comfortable using (as in the case of tariffs). I’d feel much better if there were precise tools that let the government take narrowly tailored actions. Even when one disagrees with the administration, the world doesn’t get better when a hammer substitutes for a screwdriver. The fact that regulation can be at odds with the need for responsiveness is real, but this only further highlights how important it is that effective regulatory tools be constructed now.
Instead, the structure is individual deals. If this worked, it would be one thing, but in practice, no agreement is often reached. On the Meta front, according to the NYT, “people familiar with the matter [say it’s] unclear whether they will be able to reach an agreement.”
If we are to get out of this mess, we will need better regulatory tools and structures.
Dispatch from Donald
Is the solution to the dangers of AI more AI? (It is not.)
A few days ago, Microsoft CEO Satya Nadella called for cheaper, decentralized, but no less useful AI models.
To the extent that decentralized models are harder to control, this would be bad. The Wall Street Journal’s Holman Jenkins is also concerned about Nadella’s statement, but for a rather different reason: If the frontier labs lose their customer base to decentralized and open-source models, the U.S. might lose the AI race to foreign countries. “Without customers,” writes Jenkins, “how are America’s AI leaders supposed to keep funding the push toward ever more powerful artificial general intelligence? Will IPO investors still come as expected to fill the hole?” Core to Jenkins’ thesis is the notion that AI is just another tool: The next internet, perhaps, or even the next computer, but nothing that is inherently dangerous on its own terms.
There’s no “winning” the race to build artificial superintelligence. The only winner will be the thing we’ve built. But Jenkins thinks that even the prospect of widespread job loss is “unduly speculative.” So to him, the threat of extinction is ludicrous; if one of the labs builds a model that governments can’t control, he says, then they’ll destroy that model.
(If Jenkins is reasoning from the recent shutdown of Mythos and Fable, then he’s learned the wrong lesson. Those models had capabilities that worried the White House, but they were still controllable. The worry was all about what they might do under human direction.)
Jenkins is legitimately worried about real issues, like designer bioweapons and expert cyberattacks at bargain bin rates. He cites a recent warning by the Five Eyes cybersecurity agencies, that frontier models are radically transforming cybersecurity: “The timeline is not years, it is months,” they say. But to Jenkins these are just more reasons to double down on AI R&D. Terrorists and rogue states won’t stop using AI, so we need our own AI to counter them. That is to say, the only thing that stops a bad guy with an AI, is a good guy with an AI.
But the sort of logic that grows out of gun control debates is ill-suited for AI control. Whether people are killed by guns or by people with guns, everyone will agree that guns don’t have their own agenda. (For now. I’m sure somewhere out there, somebody with an OpenClaw agent and a 3D printer is working to fix that.) AI models are different. They act autonomously. Even when they’re “obeying” you, models can go haywire and act disastrously.
His ultimate solution, not just to “bad guys with AI” but to bad guys of every stripe, is terrifying in its own right. Jenkins speaks approvingly of the role that AI can play in building an inescapable surveillance state:
In past years this column has suggested how surveillance might address America’s mass-shooter problem. The data only need to be aggregated; red-flag algorithms need to be programmed in. In the future, an irreducible function of AI will certainly be to monitor how people and governments around the world are using AI to identify and interrupt antisocial projects before they come to fruition.
This might be better than extinction, but it’s not something that I’m comfortable with, let alone a future I would welcome. Besides extinction, MIRI’s Technical Governance Team has explored a number of catastrophic outcomes from AI, including authoritarian lock-in, or the creation of a stable, global authoritarian regime. AI-driven predictive surveillance would be a step in that direction.
To be frank, I don’t think you actually have to be concerned about extinction to want to call a halt. Exhibit A in my case would be the surveillance system Jenkins anticipates. If you’re more concerned with surveillance than extinction, you should still want to call a halt: The more sophisticated the models get, the more sophisticated the surveillance can get.
Fortunately, mass surveillance isn’t necessary to halt the development of artificial superintelligence. The AI chip production chain underlying the frontier labs is centralized and straightforward to track. With sufficient political will (and it is growing), we could halt development now.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.