Dispatches from Joe
PauseCon
PauseAI US hosted a four-day conference last week in Washington, DC (4/14). They prepared a whirlwind agenda: sign-making and local organizer talks, a speaking-to-policymakers workshop, a protest outside the Capitol, and more than sixty scheduled constituent meetings with Congressional offices.
I attended in my personal capacity, both to participate and to get a sense for how PauseAI US operates. I’d never attended an event like this in my life, and felt some trepidation about what to expect, but it turned out to be both invigorating and surprisingly fun.
Artists, engineers, performers, writers, mathematicians, students, and constituents hailing from New York to Texas to Alaska all gathered to implore their representatives to speak out. I came away deeply impressed at how well the organization had managed to get so many distinct voices and values calling for the very thing our civilization most needs right now: an international agreement halting the race to build superintelligence.
They’re looking to double their number of local groups this year, and with your help they very well could. If you’d like to join your voice to those calling for a halt, you can do so on PauseAI’s website or directly here.
I wrote many more words about the experience on my blog.
Open-source catch-up?
Ron Schmelzer of Forbes argues (4/19) that open-source AI is catching up and catching on, and is now a serious competitor to the likes of Claude and ChatGPT. Open-source models publish their code and “weights” to the open internet, where others can download and (with the right skills and software) modify and fine-tune them with new data. This unfortunately doesn’t make them easy to interpret; AI is still a black box even for its makers.
Ron makes a decent case that open models and the infrastructure to run them on local hardware are both seeing a boom, but I’m skeptical the gap is as small as he implies. Open models might be only a couple months behind the leading labs in current benchmarks, but the benchmarks are kind of bad at their job. Leading models often get near-perfect scores on tests that are supposed to compare their capabilities, or else barely differ at all. And the most sophisticated models sometimes pretend they’re weaker than they are.
It’s largely good news that users have more options, but what worries me in the short (and ever-shortening) term is proliferation. I’ve written before about Claude Mythos, the superhuman hacker that Anthropic deemed too dangerous to release. Once open models reach that level of competence - which may well happen this year even if Ron is overstating his case - there’s no re-bagging that cat.
It’s how you use it
Up next is a batch of stories about accelerating AI adoption in healthcare, Hollywood, and tech, and some ways that deferring to AI can go well or very, very badly.
I expected a fairly negative framing from a piece titled “The Algorithm Will See You Now,” but was pleasantly surprised. The Wall Street Journal’s Andy Kessler writes (4/19) that Viz.ai is saving time (and quite possibly lives) in 2,000 U.S. hospitals covering 230 million people.
There’s more than one way to read that title, though. Those concerned with medical privacy might worry about AI ingesting all that data. For my part, it seems a small price to pay in exchange for shaving hours off life-saving treatment times.
By contrast, BBC Future’s Melissa Hogenboom worries that relying on AI will make doctors - and everyone else - worse at thinking. She opens with an unpublished MIT study by Nataliya Kosmyna, which measured reduced brain activity in chatbot users writing essays. I am, again, skeptical. With a tiny sample size and dubious methods, the primary study does not impress.
More importantly, though, they ask the wrong questions. The thing to care about when using AI is not how much “brain activity” you have, but whether you can spot and correct errors (or intentional subversions) by the machine, and whether the final result is actually good. An endoscopy study got closer to this important question, claiming to find that doctors who had used AI in the past found fewer tumors without it. But the study looks weak in various ways - only 14 actual cancers were found, and the study lacks a control group by design.
Ironically, Claude (the AI) spotted these issues immediately, though it was hard to interpret them properly until I dug into the study myself. Then Claude was helpful in explaining the basic medical metrics involved. This illustrates the real, and frankly fairly intuitive, dynamic at play: Current AIs can add real value as a mentor and assistant, but if you take their outputs at face value, the result is often slop.
CEO surrender?
Hollywood, of all places, might have the right idea. Forbes contributor Heather Wishart-Smith reports (4/20) on editors and directors drawing a distinction between “generative AI” (bad) and “utility AI” (good). It reads to me like the movie industry is trying desperately to deflect criticism from justifiably angry creatives with this distinction, but that doesn’t make them wrong. There’s a major difference between using AI for tedious audio editing and having it make a video from scratch.
Certain tech CEOs don’t seem to understand the distinction quite so well. Miles Klee of Wired profiles (4/20) a couple of executives who are looking to cut out the middle manager with AI assistance. Meta’s Mark Zuckerberg apparently seeks to make an AI avatar of himself available to Meta employees for one-on-one consultations. (Yes, yes, obvious robot jokes are obvious.) This seems destined to blow up in his many new faces, not least because video/audio AIs are noticeably dumber than text-only modes, but such is the fate of many early adopters.
Much more worrying is the free rein being given to AIs, and the level of deference they are likely to garner in such a regime. It was just a couple years ago that a deepfake Chief Financial Officer cost a Hong Kong company $25 million in scams. Now CEOs want to introduce deepfakes on purpose? Suppose this plan works entirely as intended, and your employees get used to taking advice and direction from AI avatars. Now the whole company is one prompt injection away from dumping its data and cash into North Korean shell accounts.
And that’s before we get into the danger of AIs with this much influence acting independently. As my colleague pointed out yesterday, a common trend in computing is that there’s not much time between AI getting pretty good at something and AI surpassing every human at it. I can already see the Onion article: Real Mark Zuckerberg ousted from Meta by army of fake Mark Zuckerbergs; “He’s just so much nicer now,” say employees.
Jokes aside, there’s a real lesson here when it comes to the future of AI. In her BBC article, Hogenboom cites a paper using the term “cognitive surrender,” an evocative term for a growing tendency to defer to AI assistants. It is contrasted with “cognitive offloading,” or “strategically outsourcing a discrete task to an external tool (e.g. using a calculator).”
Making superhuman AIs that care about humans in the right ways is an unsolved problem. You would think the AI labs would know better than to punt their most critical research to poorly-understood inhuman minds. Unfortunately, the leading labs’ explicit plan is to build AI researchers who are smarter than us, then ask them to do the work that’s too hard for humans.
That’s a little too close to “surrender” for me.
Dispatch from Mitch
AI dividend
In yesterday’s digest, I wrote a dispatch for a story about former tech workers now working as AI-regulating state legislators. Alex Bores was one of them. He’s the New York State Assemblyman running for US Congress while in the crosshairs of a pro-tech-industry Super PAC for his pro-regulation stance.
In an Axios exclusive this morning, Bores introduced an “AI dividend” plan for addressing economic shocks to US households.
Here we have, for the first time, a technology where the makers of the technology are explicitly saying that their goal is to replace all human labor.
The fact that they’ve put it out there means government needs to take it seriously.
Funded by a “modest tax on AI consumption” and augmented with taxpayer-held equity stakes in AI companies, his plan would provide “direct payments” to Americans, pay for workforce training and education, and fund “independent oversight” to help govern AI safely.
Pointing to warnings from AI companies about the transformative impacts of their products, he pressures companies with the following:
If they can support this plan, that would show that they actually believe in what they’re putting out there. If they’re not doing it, then I think it shows that they’re really putting window dressing out there.
Not a political endorsement, but this all sounds pretty sensible to me. It’s not an answer to the extinction problem, obviously, but I think job shocks could hit sooner and harder than is widely appreciated.
I will add that Bores has shown an encouraging concern for the catastrophic risks from advanced AI models “on the edge of what we can build” today. He’s heard the talk about these from the companies themselves.
It may well fall to Bores, or to whoever wins that seat, to promote legislation and international agreements to halt further development of such models before it is too late. As a bonus, I would reasonably expect such legislation to have the side-effect of moderating the economic disruptions anticipated by his “AI dividend” plan.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual analysts and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.