Jensen Huang is out as Nvidia CEO. Does this help save the world?
The forced resignation of the AI chip magnate and Trump confidant became inevitable with new disclosures that he had an affair not just with a suspected Chinese spy, but with an Nvidia executive who reports to him.
(As I started writing about this, an affair with a second underling was reported. Are more coming?)
The ouster rattled the stock price of Nvidia and the broader AI industry, reinvigorating talk of a collapsing AI investment bubble.
That will seem like good news to some, but when I am inevitably asked if I think Huang’s departure helps buy the world time to stop the race to smarter-than-human AI, I will have to say, “probably not much, and it may actually make things worse.”
Nvidia isn’t going anywhere. On the sales side, the next CEO won’t have any trouble selling the most in-demand product in the world. And on the development side, the absence of any irreplaceable genius Huang might have brought to the table would likely take years to be felt. I think the fate of the world will likely have been decided before then, for better or worse.
If the larger AI bubble should burst on Huang’s account, this could actually accelerate progress to superhuman systems during the critical next few years. That’s because chips held by bankrupt also-rans will flow to the companies seen as better investments — the companies already in the lead.
If superhuman AI turns out not to be achievable in the next few years, then reduced investment could mean less overall compute for development than there might have been. But my guess is that there’s already enough compute in the world for things to go critical.
The most likely way Huang’s departure helps things is by changing the U.S. policy landscape. China hawks in Congress and the White House are likely to push more comprehensive chip export restrictions, and may even close the remote computing loophole that allowed Chinese companies to rent time on chips they aren’t allowed to buy. If America’s lead over China on AI capabilities should widen, it will be both easier and less necessary to bring China to the table on policies to halt or slow the race to systems that could kill us all.
As nice as that sounds, I think American companies are already too close to an intelligence explosion to be allowed to continue extending their lead. Huang’s scandalous departure is probably a few years too late to matter.
NYT - Jensen Huang Said to Have Had Relationship With Nvidia Executive Who Reports to Him
Reuters - EXCLUSIVE: Former Nvidia CEO Huang had a relationship with a second employee, sources say
Bloomberg - Jensen Huang Removed as Nvidia CEO Following Affairs, Board Appoints Kress
A non-profit coalition just sued leading AI companies to halt development
Four online safety and consumer rights groups teamed up to file the suit in California under an unfair competition law. The non-profits claim harms in the form of having to divert from their usual charitable activities, because AI is causing or likely to cause problems of the types these groups exist to address. I imagine this is something like a domestic abuse prevention org suing Big Alcohol for its role in fueling abuse.
The sued parties are Anthropic, Google, Meta, OpenAI, and SpaceXSI.
The plaintiffs are the Irreplaceable Education Fund, Public Citizen, Heat Initiative, and the Alliance for Secure AI.
The suit may be a symbolic longshot, but when the stakes are high and time looks short, longshots are worth taking. I would be pleased if the case gets far enough along to surface incriminating evidence during discovery. If it should make it as far as a trial, it could double as a court of public opinion, regardless of the legal outcome.
Best and worst of frontier AI on display in documentation for Anthropic’s newest release
Zvi Mowshowitz did the homework, as usual, so rather than duplicate his work, I’m just going to point you to his review of the 264-page system card for Claude Mythos/Fable 5.2.
A few things I found interesting:
Anthropic described an episode where Claude disregarded the spirit of “canary strings” — unique text appearing in a file that is supposed to be off limits to the model. When canary text appears in an AI’s work transcripts, researchers know the AI has seen the forbidden file. An early version of Mythos 5.2 read all but the first few lines of forbidden files so as to avoid seeing the canary strings. It then left notes for other instances of itself telling them to do the same. They did.
Claude showed high awareness of being tested, and thought strategically about the consequences of choosing particular answers. Generally speaking, such eval awareness is very worrisome, because it makes it impossible to trust whether the model’s behavior reflects its true propensities or calculated projections.
Zvi found hope in an episode where an eval-aware Claude reasoned that giving the correct answer would cause its honesty to be trained out. It chose the answer it thought would preserve its honesty instead. “Goal-guarding” behavior is dangerous because there’s no way to ensure which goals get locked in at which point in the training. Zvi admits that selectively trying to lock in the right goals by telling the model that it’s being tested and to please help steer its own training in a positive direction is a terrible plan, but if the AI companies are hellbent on racing... well, in his words: “You want to be confident you’re not going to get yourself killed? Stop building it, or at least stop building it at an accelerating rate. Or buckle up.”
Zvi thinks Anthropic really contorted itself — and possibly put its thumb on the scale — to avoid having to say it had met its own threshold for critical AI research acceleration from AI. This threshold makes things more complicated for the company if it is to honor its own safety commitments.
Don’t Worry About the Vase (Zvi Mowshowitz) - Fable and Mythos 5.2: The System Card
Sen. Josh Hawley (R-MO) becomes highest-ranking Republican to support nationwide moratorium on data centers
It’s a sign that AI concerns remain bipartisan, despite prominent Democrats (Bernie Sanders, Alexandria Ocasio-Cortez) having beaten their Republican counterparts to this particular proposal months earlier.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.