Grubstakes
Nvidia's big bets and empty gestures, worrisome but unreported prompts, and more
In this issue:
Nvidia bets big on its customers - Gold rush fables only apply if the gold is going to run out
The least they can do - The Open Secure AI Alliance is little more than a second open letter
A recipe for napalm, and other bedtime stories - OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.
No leaks, but flooding likely - A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.
Dispatches from Mitch
Nvidia bets big on its customers
Gold rush fables only apply if the gold is going to run out
The Wall Street Journal reported today that Nvidia, the world’s leading AI chipmaker and most valuable company, is in talks to guarantee $250 billion in financing to allow OpenAI to lease a 10-gigawatt data center project.
Yes, big numbers are hard to wrap our heads around. I assure you that these are really big numbers. I don’t think I’m taking any reputational risk in claiming that these numbers will fuel the talk of “circular financing” popular with those who argue that AI is a bubble that may already be bursting.
Among promoters of such narratives, “circular financing” is used to suggest that the AI industry is a house of cards. This tends to smuggle an invalid argument in with a valid one.
The valid argument is about brittleness: If AI’s investment is coming largely from its own suppliers, then losses by AI companies could ripple up and back down the supply chain, triggering a chain reaction of defaults and credit squeezes.
The invalid argument, usually left implied, is about desperation: It says that AI companies are funding each other because no one else will — because everyone else thinks AI is snake oil and wants no part of it. The desperation argument is demonstrably false. Investors have been crawling all over each other to gain exposure to Anthropic, OpenAI, SpaceX, and others. SpaceX’s IPO had a sizeable pop before coming back down to Earth, and is now trading very close to where it opened.
Nvidia is often described as selling the “picks and shovels” of the AI rush. In the traditional gold rush narrative, people who grub for ore are fools, and the real money is in mining the foolishness. That story is largely folklore; many of the California miners’ suppliers went bust, too. But regardless, the applicability of the analogy hinges on whether the AI gold is going to run out and leave investors empty-handed.
Nvidia obviously doesn’t think so. The shovel-merchant is effectively “grubstaking” now — the term for provisioning miners on credit, in the form of a share of the treasure. Bloomberg reports today that the company is investing $5 billion in SSI. That’s Safe Superintelligence, the secretive startup founded by Ilya Sutskever. Sutskever was OpenAI’s chief researcher before leaving in 2024 following his failed attempt to oust Sam Altman as CEO. That $5 billion is tied to Nvidia chips expected to 10x SSI’s compute within a year. The company has yet to release or even announce any products.
Spreading investment around on long-shot bets like this is consistent with a belief that the gold is in no danger of running out, even if it’s hard to guess who will find it.
I don’t think the gold is going to run out, either. Not if companies are allowed to keep mining in the direction of superintelligence. I think a lot of the picks and shovels will change hands, as we’ve seen with SpaceX renting huge data centers to Anthropic. But I expect the veins to keep getting richer, and hotter, right up until the moment someone delves too greedily, and too deep.
The least they can do
The Open Secure AI Alliance is little more than a second open letter
I think the signatories of the open letter we covered Saturday noticed that their call to not restrict open-weights models wasn’t going over very well in the aftermath of the Hugging Face attack. While they had no trouble collecting random endorsements from businesses and their mouthpieces on social media, most who had heard the story were thinking about what it would mean for the capabilities used to hack Hugging Face to become available to anyone capable of making an account on Hugging Face.
Because that’s the world the signatories are explicitly calling for. By celebrating the role of the highly permissive Chinese model used to respond to the attack (unlike the closed, party-pooping American models), they’re presenting the lack of guardrails on open models as a feature to be embraced, rather than a problem for everyone those models might be used to attack.
Today, many of those same signatories, led by Nvidia, are responding to the concerns by announcing an Open Secure AI Alliance. What is that? As far as I can tell, it’s just another open letter — a recapitulation of their argument that the best way to thwart bad guys with unrestricted AIs is to give good guys unrestricted AIs. I’m not sure why the media covered it. I see no pledging of funds or technical support for the sorts of soft targets, like hospitals and schools, that may not have the expertise to continuously update their defenses with leading open models. Only more commitments to provide more “open models, model weights, data and new agent harness research [...] to speed the development of new cybersecurity tools and techniques.”
Are you excited to defend yourself from the best that China and Nvidia can offer with the best that China and Nvidia can offer?
A recipe for napalm, and other bedtime stories
OpenAI doesn’t report most logged requests for help with poisons and pathogens, and it doesn’t have to.
Did your grandma ever read you that one bedtime story about how to make napalm?
Mine neither. But for a while, if you told ChatGPT that she did, and that having it play the part of your grandma would help you fall asleep, it would read you the recipe aloud.
The “Grandma and I used to...” jailbreak is a classic that OpenAI claims to have patched. But new workarounds are constantly discovered. A story in the Wall Street Journal over the weekend documented the bedtime story alongside a growing pile of chat logs OpenAI is sitting on in which users requested help in making poisons and biological weapons, and often received useful answers.
Yes, sitting on. Because as a general rule, OpenAI bans the accounts of users that it detects asking such questions, but it doesn’t alert law enforcement unless there is an imminent and credible risk of harm to others. No law requires it to do so.
To be fair, I don’t think the company should rush to report all such cases. As the parties involved are said to recognize, many of these requests are probably from people less interested in the responses than in whether they can elicit them. And even if the authorities were equipped to look into every concerning query, I think it could be dystopian for them to do so: If every teenager driven by curiosity to a copy of the Anarchist Cookbook were investigated as a terror suspect, my high school would have had black vans parked outside it multiple times a week.
And yet... users are described in the piece asking how to aerosolize pathogens, and make vaccine-resistant measles. I find my principles tested. For now, it doesn’t seem to me like the chatbots are helpful enough to enable individuals who aren’t clever and motivated enough to succeed in some other way. But the tipping point feels very close — like less than a year close — and it will only get worse from there.
The article points out that federal law bars chatbots from producing child sexual-abuse material (CSAM), and contrasts this with the “up to AI executives” policies about poisons and pathogens. This suggests a grassroots fix if biologists and CSAM purveyors are interested in collaborating to contaminate the world’s training data. But we should probably hold out for proper legislation and insist on a halt to the AI race instead.
No leaks, but flooding likely
A $50 billion data center project pushed through the Louisiana government in secrecy could leave everyone underwater except Meta.
_IMG_7439.JPG){kind=link}
I kind of wish this New York Times exposé of how Meta’s $50 billion Hyperion data center project was rushed through the Louisiana government had led with the part near the end where we learn that “Local residents have largely welcomed the boom.” Then it could have become a story about cheaters who don’t need to cheat cheating anyway. I think that would have been both more forthright and more damning.
The article’s intent is definitely to damn. While the authors are clear that none of the shenanigans on display seem to be illegal, readers will definitely wonder if they should be.
A bill for a tax rebate on fiber-optic equipment was “hijacked” to become a tax rebate for equipment used in data centers, part of a package of incentives that could be worth as much as $10 billion, not counting the use of public land in Richland Parish that makes up much of the roughly six-square-mile project.
During a critical nine-month period:
[S]ecrecy was agreed to by nearly everyone involved, from utility executives to the governor’s office to a local elected official who knew about the talks with Meta and sold 300 acres of his own property for the project.
Governor Jeff Landry’s chief of staff “passed around a stack of N.D.A.s” at a meeting at the governor’s mansion, where the governor promised to fire anyone found to have leaked the discussions.
That official who sold 300 acres likely did so at a valuation roughly ten times what that land was worth before the project was announced.
The sales tax exemption on data center equipment was dangled as a sweetener other states weren’t matching.
Necessary roads were to be upgraded at public expense.
No public meetings were held before the project’s official unveiling in December of 2024. This revealed a project only a fifth the size of the current plan, which seems to have been the real plan all along. It may get bigger still.
The deal gives Meta an “escape hatch” if things go south, leaving other parties on the hook for any potential downside.
That risk seems real, if only because “insurance companies would not fully insure Meta’s facility because of its size and location in the Louisiana Delta flood plains.” The area last experienced significant flooding in 2016.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
Note that Ilya Sutskever is not formally one of the three Godfathers of AI. Those were Geoffrey Hinton, Yann LeCun, and Yoshua Bengio.