In this issue:
It’s only creativity if... - Moving goalposts put VFX artists on wrong side of “creativity” divide
Pass me that splewdriver? - A new type of cyber attack exploits the hallucinations of coding agents
Dispatches from Mitch
It’s only creativity if...
Moving goalposts put VFX artists on wrong side of “creativity” divide
The easiest way to keep claiming that AI isn’t creative is to define creativity as whatever AI can’t do yet. The trouble with that definition is that it increasingly puts humans on the wrong side of it.
We live among AIs that can write 10-page epic rhyming poetry, about a haircut, where every word starts with the letter s. (See this post from last month for context.) We live among AIs that can independently beat the daily challenge in the game Slay the Spire 2. As Ethan Mollick, the researcher who posted both those results, put it:
Slay the Spire 2 is a very complicated game, with lots of choices that have large-scale strategic implications that may not pay off (or doom you) for many moves, along with careful resource management. And it was released post-training period.
If you told a human who had just pulled off either of these feats that they hadn’t actually done anything creative, I think they would beg to differ.
If pressed, some might say that humans solve these problems with creativity, while AI solves them in some other, more mundane way. But at that point, it’s unclear whether the word “creativity” is anything more than a brand name, like how it’s only champagne if it comes from the Champagne region of France.
If the point of the branding exercise is to protect the livelihoods of human creatives as if they are French vintners, I would argue that it’s already failing.
There was an article in the Hollywood Reporter a few days ago about how Netflix bought Ben Affleck’s AI visual effects company for $587 million in cash. My eyes popped out at the amount, but I wasn’t planning to write about it because the purchase itself was old news, and it looked like a pure business story. But then I saw where the journalist described the company as having a “particular emphasis on upending visual effects without impacting the creative part of filmmaking.” (bold mine)
VFX isn’t one of the creative parts? I thought maybe that was just the reporter’s gloss, but a blog post by Affleck states:
I knew I had a responsibility to my peers and our industry, to protect the power of human creativity and the people behind it.
This “human creativity” apparently does not extend to the work Affleck says they trained their first model to do:
[It was] trained to understand visual logic and editorial consistency, while preserving cinematic rules under real-world production challenges such as missing shots, background replacements or incorrect lighting.
So yes, the claim is that achieving aesthetic effects under challenging real-world constraints doesn’t require creativity. Got it. Has anyone told this to actors, directors, and film crews wrapping up a tough shoot?
The moral of the story: Insisting that AI can’t be creative doesn’t keep it from coming for your job. It just causes people to decide your job was never creative to begin with.
This pattern isn’t new, by the way. It goes back to well before the ChatGPT era. Check out the latest video from Rational Animations for a history of the moving goalposts around AI creativity.
Pass me that splewdriver?
A new type of cyber attack exploits the hallucinations of coding agents
A classic scheme of the early internet era was “typo squatting”: buying a website domain name that people might type by mistake. A page like “yagoo” or “myspage.com” could be used to serve malware or fool a user into entering their Yahoo or MySpace credentials.
An AI-era twist on this scheme is the so-called “HalluSquatting” attack, the recent subject of a Fox News column by Kurt Knutsson. This attack exploits the fact that AI coding agents sometimes hallucinate — the “hallu” part of HalluSquatting — and do so in predictable ways. Because today’s AI agents lack shared and long-term memories, it’s like every new session of Claude or ChatGPT is waking up at the start of a time loop. This means they may unknowingly stumble into some of the same mistakes they made the last time around, if put to work on a similar task. Some of these mistakes involve hallucinations, like reaching for coding tools that don’t actually exist.
As an analogy, imagine that you have a prolific assistant in your workshop who just used your bandsaw, lathe, and drill press before reaching for a splewdriver. Not finding one, they tell you, “Hey, I notice you don’t have a splewdriver. It’s the right tool for this job. Mind if I order us a splewdriver?”
You scratch your head, because you’ve never heard of a splewdriver. But your assistant is really skilled at their job, so you assume they know what they’re talking about and say, “Sure.”
What happens next? Ideally, they fail to order a splewdriver, because the tool doesn’t exist. But what if it does? What if someone made a tool called a splewdriver and listed it in a marketplace for your assistant to find? Now you’re in trouble, because that tool could be anything.
This is basically what can happen to coding agents making use of open source code repositories. They will sometimes hallucinate and try to download code libraries that shouldn’t exist. Bad guys who have studied those agents make those libraries exist. Coding agents download those libraries, installing malware or passing vulnerabilities into their users’ new code for bad guys to exploit later.
This attack can carry much less friction than in my workshop analogy. Users who tire of approving every little action can configure their agents with blanket permission. A malicious package an agent reaches for probably doesn’t have an obviously suspicious name like “splewdriver” that would cause an attentive user to perk up. And the download is probably free.
A research paper on HalluSquatting found hallucination rates as high as 85% in scenarios where agents act to download code libraries. They also found that AIs from different companies sometimes hallucinate the same non-existent tool names, which could make squatting attacks on those names especially lucrative.
It’s not clear how widespread these attacks are, but they are proven to work. For now, the only real defense is the obvious one: Don’t let your agent download software for you, and only use code libraries you know and trust.
My worry is that many of the people now getting into vibe coding lack even a minimal coding background, and so might not even know what a code library is. If the word “bandsaw” is just as alien to you as “splewdriver”, you’re probably just going to let your assistant run the shop.
There’s a larger lesson here about the difficulty of trying to oversee the work of an expert that outclasses you in a given domain. The AI companies hoping to keep superhuman AIs in check by leaving their designs to slightly less superhuman AIs are racing to be in a position where everything sounds like a splewdriver to them.
If we let them do that, we’re splewed.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.
I couldn't move on without complimenting the wit in the last sentence!