Dispatches from Joe
How can we retain the ability to oversee AI?
A World Economic Forum article today observes that on many doctorate-level questions, “machines have moved from well below expert level to well above it” and points out some problems this situation poses for oversight of AI. For my part, I think they are correct to worry, but still missing the bigger picture.
The article spotlights three issues. First, people tend to trust automated systems that are usually right. Second, delegating decisions makes it hard for would-be overseers to get the hands-on practice they need to distinguish good actions from bad. And third, the law can require human oversight, but it can’t make the overseers competent.
The authors recommend structured practice for knowledge workers, AI literacy for advisors, and a demonstrated ability to override the system. This is all well and good, but...I’m still concerned.
In my previous career, I’ve used (and built) automated systems that ease the burden of mind-numbing tasks. I notice the authors’ advice seems pretty good, if you’re using a system built before modern agentic AI. But I’m not sure they fully grasp what the last four years have wrought.
I think it’s optimistic of them to assume there will be a human in the loop at all.
Many early AI agents routinely stopped what they were doing to ask the user for permission. Is it OK to run this console command? How about this one? Pretty please may I search the internet? Users quickly tired of mashing “yes.”
The solution? Automate the approval. Some premium AI agents, like Claude Code, have an “auto mode” in which another AI reviews the requests from the first AI and automatically approves the ones that seem innocent. Which is, in my experience, nearly all of them.
Why wouldn’t people use this? It’s an enormous quality of life improvement. An AI agent running autonomously can build an entire website in the time it takes a human to review and approve a single complex command. The ask-for-permission method is agonizingly slow by comparison.
I want to stress that most of the time, this process works completely fine. Modern AIs are startlingly competent. Anyone who isn’t a domain expert is heavily incentivized to just let the AIs do all the work, and the domain experts themselves are increasingly managing multiple AIs at once. As for the risk that the AI will delete your customer data and all its backups in nine seconds flat — well, humans sometimes do that too.
The problem is that this generalizes to oversight as well. When human input actively hinders the productivity of an AI, there will be immense pressure to skip the approvals or hand them over to yet another machine.
This is more or less how AI companies say they plan to make their AIs care about human flourishing. The problem is too hard for humans, so they plan to build AIs smarter than they are and make the AIs do it. This plan has all the oversight problems the WEF article points out, and more: how do you oversee something that’s more competent than you in every way that matters?
With any other technology, we might be justified in letting the reckless take their lumps and learn by trial and error.
But AI companies have set their sights, not on dumb workflows, but on entirely new minds which are smarter and more capable than humans. It’s extraordinarily dangerous to build such minds, and doubly so to depend on them. If the AI race continues, companies and even governments could lose the option to oversee AI at all.
A fresh start for federal AI regulation?
After lifting the ad hoc restrictions on AI company Anthropic, the administration has an opportunity to reset its policy on frontier AI. The Financial Times reports that the White House is in talks with Anthropic, OpenAI, and Google to work out a consistent set of standards for potentially dangerous AI models, and that a new set of rules could be ready as soon as next week.
These talks come exactly halfway through the 60-day window given by last month’s executive order for federal agencies to work out a framework for evaluating frontier AI. Given the speed at which AI companies are currently racing to enhance their models, the government can hardly afford to dawdle. But I do hope they have time to develop (or borrow) an effective framework. Good governance needs far more visibility into dangerous AI capabilities than the public currently has.
Dispatches from Beck
Economists consider AI a bet with unknown odds
As a general rule, economists used to be very skeptical of AI’s effects on the economy.
That is no longer quite so true, but despite that change, neither economists nor the data they cite agree on much. That even includes basic facts like the current effects of AI on jobs, the New York Times reports. One economist, Michelle Yin of Northwestern University, complained that “it’s like going to the doctor and getting three different diagnoses for the same condition.” I think that means the responsible thing to do is first, get better data, and second, not just assume the diagnosis you’d prefer is correct.
So let’s examine these different diagnoses. Some report significant effects on entry-level white-collar jobs. Among them is Stanford’s Digital Economy Lab, whose dashboard shows significant decreases in entry-level jobs exposed to AI. But, zooming out, while unemployment of young people is going up, it remains lower than it was for most of the 2010s, particularly for those without degrees. And AI exposure is itself very sensitive to how said exposure is examined, with one study showing that changes in study design can give significantly different measures of job availability.
In another genre of diagnosis, Bank of Canada Governor Tiff Macklem, quoted in the Wall Street Journal, worries about the tremendous pace of current investments, which only pay off if companies continue to succeed. He said that long term promise doesn’t rule out a crisis: “The internet proved to be better than anybody imagined, but we still got the dot-com bubble.”
And a third category of diagnosis, from those like the new Federal Reserve Chairman Kevin Warsh, says that AI will positively reshape the economy. Warsh argues that AI will have deflationary effects that justify cutting interest rates now. Put (over) simply, AI can improve the productivity of workers and companies, reducing the cost of their outputs. With this downward pressure on prices the Fed can lower rates to boost demand without much risk of reigniting inflation.
Warsh anticipates these increases in prosperity and said, “We are in the first or second inning of this revolution. If you wanted me to sound like a pessimist and a doomer on this, I’m afraid I’m not there.” I don’t want to nitpick on terminology too much, but by collapsing ‘doom’ into ‘there could be job losses’, he illustrates how even those who think AI is revolutionary are not taking the full range of possibilities seriously.
And, finally, some think we might be in a lose-lose scenario. Torsten Slok, chief economist at Apollo Global Management, notes that AI spending is a huge bet; according to Slok, 87% of all venture capital funding this year is AI-related. He argues that if “AI is wildly successful,” then it will replace jobs and drive up unemployment, ultimately undermining consumer spending and leading to recession; if not, then today’s investments will go bust, seriously weighing down the economy.
Thankfully, the government is beginning to take action to address this uncertainty. Senator Mark Kelly, in introducing a bill to expand data collection, said: “The government’s got to make some big decisions about A.I. and about the economy, and if you’re doing that in a vacuum, you’re going to make mistakes [...] you can’t do this smartly without reliable data.”
I think that uncertainty about effects, particularly about the pace of progress, is deeply reasonable, but I don’t think that means that the world’s response is also reasonable. Even setting aside risks of extinction, while transformative effects are in the mix, the correct response is not to assume you’ll get what you want — uncertainty should inspire caution, not speed.
Right now, a small number of companies are making a bet on behalf of humanity; even if you like their odds, we shouldn’t let them bet the house.
Consciousness studied
In another sign of the changing times, AI consciousness research is now widespread, the Washington Post reports. Cambridge Digital Minds, a new academic research program, is one of at least 46 organizations studying the topic, counted in an online guide it helped assemble.
Reading through the guide, it’s interesting to see blogs like Astral Codex Ten and Don’t Worry About the Vase covered as authoritative media on the topic alongside the research of multibillion-dollar companies like Anthropic and Google DeepMind.
Openness to the possibility of AI consciousness is somewhat rare among figures with established clout (you may recall my colleague Mitch’s coverage of author Ted Chiang’s dismissiveness of AI consciousness here).
But the times keep changing, and, as Cambridge assistant professor Lucius Caviola said, “This issue is becoming less and less weird.” Let’s hope the range of academic curiosity keeps expanding to other pressing AI topics.
Dispatch from Donald
OpenAI offers a 5% financial stake to the U.S. government
Last month Senator Bernie Sanders proposed that high-revenue companies in the AI industry pay a tax in stock. Now, OpenAI has reportedly started a conversation with the Trump administration to give the U.S. government a financial stake in the company.
In both cases, the government would acquire a financial interest in the company. (The “tax on stock” that Sanders proposed is not like a “tax on cigarettes,” but more like a “tax paid in stock.”) There are some key differences between the proposals: Sanders’ plan would apply to every company that reaches $200 million in annual AI-related revenue, and deposit 50% of each company’s stock into a public fund managed by an independent commission. (The company would have to issue new stock to pay the tax.) What OpenAI is proposing is voluntary and limited to a 5% stake.
Another key difference between the two approaches: OpenAI is proposing to hand over a share of future profits; Sanders wants votes and seats on the board of directors. If I’m reading the proposed bill correctly, Sanders’ proposed fund would hold exactly 50% of an affected company’s stock. Were the company to issue new stock in the future, some of it would have to go into the fund in order to keep that 50/50 ratio.
I’m wary of giving the government a stake in any of these companies. Policymakers don’t need another reason to be reluctant about restraining frontier research. But of the two, Sanders’ proposal has at least one thing going for it: An independent commission is a lever by which the public can slow development and veto dangerous decisions. (It’s not the best lever — I would much prefer an international treaty — but it would be an improvement on what we have now.)
If the Trump administration agrees to OpenAI’s plan, this won’t be the first time it has acquired a stake in a company: In 2025 the U.S. government acquired a 10% stake in Intel. In any case, as The Financial Times notes, any deal may have to pass through Congress.
This isn’t the first time that OpenAI has floated the idea of some kind of government stake or public wealth fund. It might be an intentional attempt to compromise regulatory decision-making, though good old-fashioned lobbying often does that job. If I had to guess (and this might be a little reductive, because humans are complicated enough to have multiple motives), the point is to launder public opinion. Many people don’t like AI companies, and that dislike seems to be growing stronger and more widespread as AI becomes a more prominent issue. “Please like us; we’ll give you a few crumbs from the table” may not work out as well as they hope, but I can see why they’d try it.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.