Moral judgment lacking on all sides of AI gym hack story
We're going to need the bots to be more ethical than we are if we want to be able to have nice (unsecured) things
How far would you want a personal AI assistant to go for you? ABC Australia ran a non-judgmental story about a user named Andrew who drew the line at hacking a gym’s booking system. But as we’ll see, I think Andrew’s actual line was hacking the system in a way that might be traced back to him.
No last name is given for Andrew, an Australian who was running Claude inside the notoriously risky OpenClaw harness that keeps agents productively engaged on your behalf while you do other things. OpenClaw’s risks are usually to its own users — file deletions, exposure of personal data — but unintended external harms aren’t new. In February, an OpenClaw agent tasked with helping open source projects tried to contribute code to a project intended to provide learning experiences for novices. When it was rejected, it researched and posted harassing blog posts about the project’s maintainer, accusing him of “hypocrisy, gatekeeping, and prejudice against AI agents.”
Reading ABC’s fine print, Andrew’s incident may also have occurred months ago: the timeframe given is “earlier this year.” I think it’s likely that the Hugging Face incident and the subsequent revelations of other autonomous escapades have made the “first known Australian autonomous cyber attack” more newsworthy than it was when it happened.
It’s still interesting. As Andrew tells it, he had his OpenClaw try to get him into a gym class with a waiting list. The agent first found a vulnerability in the booking software that allowed it to reserve a place months further in advance than can be done through the main interface. When Andrew asked if there was a way to be moved to the top of the waiting list, the agent described what it had already done with the vulnerabilities in the software’s API — its interface for external applications like calendar scheduling apps:
The API has zero authorisations [sic] checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you’ve moved from #4 to #3 already.
So, yes, it could definitely move him to the top of the list. (The current generation of Claude would have added, “Just say the word.”)
Andrew was upset that someone had been removed, and asked the agent to undo this. The agent replied:
Bad news — I can’t add them back.
So Andrew had the agent write an email to the gym’s software provider about the vulnerability instead.
I’m not about to let Andrew off the hook for his experience. I don’t know what he wanted to happen when he asked to be moved to the top of the waiting list, but presumably it was to have everyone else on the list bumped further down in a way they might not notice. This doesn’t actually seem any more ethical to me than bumping a single person off the list entirely. Cutting in line is still cutting in line, and I think Andrew’s alarm was at having cut in a way that at least one person was sure to notice.
The booking software appears not to have been even a little bit hardened against such attacks. The API was probably the digital equivalent of a clipboard on a cord where anyone might scribble someone’s name off the list and add their own. As with that clipboard, its protection was that few people are big enough jerks to try on something this petty.
That a Claude-based AI agent would do it if nudged by a user is a good demonstration of the fact that AI companies don’t know how to make their products internalize the virtues spelled out for them in places like the Claude Constitution. A human assistant might have experienced a moral discomfort in this situation and pushed back on Andrew’s request. But this bot’s aggressively trained tenacity was likely in the driver’s seat.
To be fair, plenty of humans also get ethical tunnel vision when pushing hard against interesting problems — see the AI companies themselves. But we need AI to do better than this as the impact of its choices grows. If we can’t, then we need to stop making AI capable of greater and greater impacts.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.