For as long as I’ve been using AI, or even the internet, I’ve assumed that people and companies could pinpoint my location and identity if they were really trying. There are myriad ways to identify me, from tracing my IP address to using AI to recognize my writing style.
It turns out this was a little bit pessimistic. AI companies do keep records of accounts and IP addresses that access their systems, and records of the conversations those accounts have with company AI, but they don’t generally take fine-grained identification data from a user’s own system. China recently accused Anthropic of breaking this unwritten rule by quietly storing GPS and other identifying information belonging to users of Claude Code.
This was most likely an attempt by Anthropic to catch illicit use of its AI, like the mass cyber campaign it caught last year, or the “distillation” of Claude’s capabilities into cheaper AIs, or the Chinese black market in cheap access to Anthropic’s models. It’s an understandable move, and not exactly the “serious threat” or “back-door vulnerability” that China would presumably like its citizens to believe. But as far as I can tell, it was unannounced (not mentioned in the changelog) and done in a plainly sneaky fashion.
In a move that sure looks like an attempt to obfuscate the data transfer, location data was encoded in tiny details in the system prompt (the instructions sent to every instance of an AI). One of these details was the date separator (either a hyphen or a forward slash) and the other was the exact kind of apostrophe in the phrase “Today’s date is...”
It should not be news to us that AI companies can, and sometimes do, secretly modify user requests, or store user data. Remember: they keep all your conversations. It says so in the privacy policy! Chat transcripts can even be used as evidence in court.
Some sources claim the hidden tracking has since been removed. Users still expressed concern over the subterfuge, and I don’t blame them. This incident looks to me like Anthropic is flying by the seat of its pants regarding security, and it doesn’t leave me very confident in their ability to protect the data they store.
In many ways, privacy trades off against catching illegal and unethical behavior. I don’t pretend to know exactly where the line should be. But I do know this: if you were previously under the impression that AI companies are being open and responsible about their use of your data, today`s a good day to reconsider.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.