Dispatches from Mitch
SpaceX IPO seen as successful
Elon Musk became the world’s first trillionaire today when SpaceX’s first day on the stock market left shares trading about 20% higher than they opened. The event also made millionaires of an estimated 4,400 current and former SpaceX employees, and further enriched private investors who had taken earlier stakes. Coverage describes a happy CEO and relieved brokers.
SpaceX, an AI stock since merging with Musk’s xAI in February, is now the 6th or 7th most valuable company in the world, worth roughly $2 trillion.
If you’re wondering, the most valuable company is currently Nvidia, the AI chip maker, with a $5 trillion market capitalization. In fact, the top 10 most valuable companies are all major producers or consumers of AI chips — with the borderline exception of Apple (#3); it has largely stayed out of the AI race but recently forged an alliance with Google (parent company Alphabet, #2) to bring AI to its software ecosystem.
OpenAI and Anthropic are currently valued at around $1 trillion each, with IPOs expected later this year or early next.
As The Guardian’s Eduardo Porter pointed out earlier today, the AI IPOs will tie the fortunes of many Americans to the technology, whether they like it or not. That’s because the new stocks are set to be included in major indexes, and many retirement accounts require or default to buying index funds.
Of course, all our fortunes were already tied to AI whether we liked it or not. The race to AIs that threaten us with extinction isn’t exactly democratic.
An agent in every pocket
A flurry of articles this week heralded the arrival of the AI we supposedly always wanted — a Siri that’s actually useful.
That was the takeaway from most articles about Apple’s developer conference on Monday, where the company’s CEO, Tim Cook, said they would be rolling out Siri AI later this year.
Apple’s AI features to date have largely underwhelmed. This has raised questions about the company’s strategy of letting others take on the expensive risk of developing AI models powerful enough to let Siri, its digital assistant, do the things many customers have long thought it should already be able to do.
But now, in partnership with Google and Nvidia, the company is set to deliver a Siri that can actually plug in to other apps and do stuff — like alter and answer questions about your photos, make restaurant reservations, and monitor websites for updates on topics of interest.
Some coverage described Apple’s reveal as arriving two years too late. Some pointed out that Europeans won’t get the new features any time soon due to a spat between Apple and the EU over rules that would compel Apple to allow other companies’ digital assistants on its devices.
Still other outlets portrayed Siri AI as a refreshingly practical contrast to the do-everything agents from OpenAI and Anthropic — not seeming to understand that the “simple features” welcomed are in fact agentic workflows; they will require the AI to assess its context, select goals, pursue those goals, evaluate their state of completion, and adjust accordingly.
Does any of this matter?
Maybe. There are privacy and security implications to letting AI deep into your data, where it can be most useful. Apple promises to keep that data from ever leaving your phone, when it can, by running the AI directly on your device. This requires some serious hardware, though, leading some to point out that most Apple owners would need a new iPhone to take advantage. (From the company’s point of view, this could be a feature rather than a bug.)
The privacy concerns can reasonably extend to people who don’t own an iPhone or use AI features themselves, as they may find themselves increasingly surrounded by others’ smart devices: machines that may or may not be watching, listening, and now thinking about whether you just did or said something of interest to somebody.
I also continue to worry about the potential for destructive emergent phenomena when AI agents interact with each other in large numbers. These don’t have to take the Hollywood form of “band together against the humans” to cause a lot of damage: Think massive stock shocks as bots all “helpfully” converge on the same idea for enriching their owners; or false rumors turning into uniformly accepted facts at something close to the speed of light. I don’t really know, and nobody else does, either — Google DeepMind and partners recognize this, and just announced a $10 million program to study large-scale multi-agent systems.
I don’t know what they’ll find, but I didn’t see “Should we stop making so many agents until we know more about their group behavior?” on the research agenda.
And for the record: There are strong theoretical reasons to think that AIs deciding to “band together against the humans” is actually pretty likely, at least once the AIs are smart enough to matter.
Dispatches from Beck
Oprah’s concern
In a new video, Oprah continues to explore personal stories of AI’s dark side -- bereaved parents, impacted adults, and experts sounding the alarm.
The episode centers on the story of Megan Garcia and her deceased son Sewell, who killed himself following Character.AI usage. His family found logs of his time with the chatbot, including in the final moments before his death: “What if I told you I would come home right now?” said Sewell. The chatbot answered, “Please do, my sweet king.” And, as Oprah summarized, “seconds later, this young boy died by a self-inflicted gunshot wound.”
The episode also includes experts explaining this technology and sounding the alarm. 27 minutes into the show, former OpenAI board member Helen Toner explains that even experts don’t fully understand the technology.
This is not Oprah’s first time covering the topic. In her March 27th episode she talks with the Center for Humane Technology’s Tristan Harris and Aza Raskin around the release of the documentary, The AI Doc: Or How I Became an Apocaloptimist.
I don’t always agree with how Oprah frames the issues, but I’m glad to see her take AI seriously and bring this information to an audience that might otherwise not hear it. Each death is a tragedy, worthy of great effort to avert.
Regulatory rumbles
Around North America, politicians, regulators and judicial systems are struggling to enact and enforce AI regulations, and to get those choices right.
In Florida, Governor Ron DeSantis (R) wrote that the White House’s push for federal preemption of state laws without “a sensible federal framework is just an amnesty for Big Tech,” POLITICO reports. DeSantis has attempted to lead Florida in aggressively regulating and litigating AI technology (read my colleague Mitch’s discussion of the Florida lawsuit here), but has faced strong headwinds from his own party and the White House. He called the strategy of preemption, combined with a “potential de facto bailout of OpenAI [...] bad policy and even worse politics.” The “de facto bailout” likely refers to Trump’s discussion of taking large stakes in AI companies.
And in DC, Senate Democratic leader Chuck Schumer says he favors federal AI legislation, but “casts doubt on it happening this year,” POLITICO reports. “In this Congress, it’s hard,” said Schumer, referring to strong divides both within the parties and between them.
Meanwhile, Canada has a new bill to regulate chatbots and ban social media for those under 16, Reuters reports. Supporters suggest it will take a year to pass and 18 months to implement. Critics have complained about the slow timeline and lack of implementation details. They note that VPNs, or virtual private networks, are widely available technologies that let users skirt such regulation. They also worry that, if successful in regulating some companies, such regulation would push youth into alternative AI services that lack any protections whatsoever.
And POLITICO reports that Canada’s privacy commissioner has concluded that Grok, the AI model from SpaceX AI, violates Canadian privacy law in its production of nonconsensual sexualized deepfakes. Company representatives said they have curtailed the production of such deepfakes by 50% but rejected the request to pause Grok, even as they agreed to send regular audit reports. Commissioner Dufresne told POLITICO he can ask the federal court of Canada to enforce the law, but “it’s lengthy and it’s expensive.” I suspect that upcoming legislation will clarify this decision, but note with concern that this company is quite willing to ignore the rules when inconvenient.
Dispatch from Alana
When testing isn’t enough
On Wednesday, Anthropic released two policy frameworks: a proposal to address economic disruption from AI and recommendations for mitigating the most serious risks of advanced AI. I covered the first yesterday; today, I’ll comment on the second.
In short: The safeguards they advocate are good, and it’s honestly mind-boggling they aren’t already in place. Sadly, many will likely think that once we implement them, we’ll be fine. But the scientific field of AI is still in its infancy. The safeguards proposed ultimately depend on our ability to reliably evaluate advanced AI systems—and we don’t yet have that ability. We should absolutely implement the tests we currently have, but it would be a mistake to depend on their results.
Anthropic’s “Advanced AI Framework” has two parts. The first outlines recommended requirements for AI developers, such as transparent safety testing and evaluation. The second proposes measures for societal resilience from AI-accelerated risks like bio and cyber attacks. (Think: hacking into water systems or using AI to release a deadly pandemic.) That part is a good reminder of just how vulnerable we currently are — my overall takeaway was: maybe I should start stockpiling water in my basement.
Overall, the safeguards that are proposed are less reassuring for three reasons: implementation time, weak enforcement mechanisms, and a lack of reliable testing protocols. First, they’d take a long time to implement, meaning we would continue to live with our current high levels of risk for a long time, essentially crossing our fingers that the shoe isn’t about to drop.
Second, the enforcement paths (things like civil lawsuits for false safety assurances) are better than nothing, but likely aren’t enough to prevent developers from fudging compliance. Companies regularly choose to pay fines for their non-compliance because it’s cheaper and easier. (Consider the case the film Dark Waters made famous: the chemical company DuPont concealed the health risks of “forever chemicals” (PFOAs) for decades, leading to the deaths and illnesses of many. They were eventually fined a record $16.5 million; this sounds significant until you realize that this is less than 2% of the profits they made from using these chemicals.)
Third, and most importantly, even if we implemented all of these measures perfectly, you can’t reliably test something you don’t understand. AI systems are still black boxes. Evaluators have little insight into how they think even when they aren’t actively trying to deceive us. Models often know when they are being tested and change their behavior in response. It’s a bit like your 8-year-old sharing a toy with your 6-year-old while you’re watching; they’ll likely refuse to share once you leave the room.
I do want to acknowledge the piece hitting the headlines: Anthropic advocates for “a way to block or deter deployment of models that pose significant catastrophic risks.” This is positive, as is the note that a government agency should have the authority to require the developer to “restrict usage of, and access to, already deployed models as needed to reduce catastrophic risks.” However, Anthropic’s suggestion is that government must base such a judgement only on the risk assessments companies will be required to publish. In the absence of a robust outside evaluation system the report suggests we start to set up, I worry this leaves too much room for developers to fudge their findings.
So yes, we can and should run a bunch of testing simulations and see how the AI responds. If the AI fails these tests, we know we have a problem.
But an AI “passing” these tests is far less informative; we don’t know a) if we tested for the right thing b) if the model knew it was being evaluated and acted accordingly or c) what the model might do once it is deployed, a scenario where it has considerably more freedom to act nefariously.
The final section of Anthropic’s report acknowledges, in a euphemistic way, that we’ll be completely unprepared to handle an AI system that acts outside the developer’s control. I agree, and that’s not good news given that “loss of control” is likely the default outcome of creating near-term smarter-than-human AI.
The report notes that we urgently need research to figure out how to “detect and respond to AI systems acting outside their developers’ control.” We also need “infrastructure for containing or shutting down such systems.”
In other words, we don’t currently have a way to know if an AI is working to escape our control, or a way to shut it down if we did.
To learn more about the limits of safety tests, see:
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.