In this issue:
Lawmakers look to curb Chinese AI - Critiquing some claims about American and Chinese models
Today’s data - Claude Code secretly tracked user data
65% of Americans are concerned about AI - Results from the tracking poll “Americans on AI”
Dispatches from Joe
Lawmakers look to curb Chinese AI
Critiquing some claims about American and Chinese models
U.S. lawmakers have noticed that Chinese AI is growing more popular, CNBC writes.
I’m glad to see that two House committees have begun asking questions, but I saw several claims in the coverage that made me wince. I’m going to take this opportunity to set some facts straight about American and Chinese AI.
Andrew Garbarino, chairman of the U.S. House Committee on Homeland Security, remarked to CNBC:
Recent reporting that a Chinese open-weight model can match leading U.S. models in certain vulnerability discovery and cybersecurity tasks is highly alarming.
This reporting is largely false, and I’m sad to see a sitting House committee chair amplifying it. No, Chinese models are not as good at cybersecurity as Anthropic’s Mythos. They are, however, much cheaper to run, and adequate for many tasks.
This cost advantage led Andy Ogles, chairman of the Subcommittee on Cybersecurity and Infrastructure Protection, to observe:
If we do nothing, Chinese models become the default foundation of the global digital economy, carrying embedded censorship, uncertain security, and capabilities distilled from our own laboratories with the safety guardrails stripped out.
I give Ogles two point five out of three for accuracy. Not bad. Chinese models do carry censorship; try asking DeepSeek about the Tiananmen Square massacre.
It’s worth noting that U.S. models do some censorship too. Grok at one point seemingly censored negative coverage of certain leaders, and Google’s Gemini was once documented refusing to answer “where is Palestine?” Various kinds of soft censorship and politically noncommittal behavior have featured in the news as well. But I expect significantly tighter and more consistent censorship from China.
Is the security of these models “uncertain”? If anything, that’s an understatement. But security concerns are more a feature of AI in general than Chinese AI in particular. No company, American or Chinese, knows how to exert robust, fine-grained control over the alien agents they’re building. That said, if you access a Chinese AI through a Chinese server, you should probably assume that the whole conversation can be seen and influenced by someone in China.
What about the final claim, “capabilities distilled from our own laboratories with the safety guardrails stripped out”?
This one is also true but slightly misleading. Distillation, or using the outputs of one AI to train another, is common practice in industry. American companies do it, too. When they release a new open model, sharing the AI’s weights so that anyone can run it, there are pretty good odds that model is at least somewhat distilled from Claude or ChatGPT.
An unnamed aide mentioned one proposal being considered: boosting open-weight American AI models, so companies aren’t tempted to turn to China. I think this misses the point. Users can strip guardrails from any open models, not just Chinese ones. If Meta releases an open model that can develop deadly new pathogens, bad actors can easily crack its safeguards, and that model is just as dangerous as a Chinese model with the same capabilities.
What should be done instead? I suspect that the growing popularity of Chinese AI is partly a result of recent American policy errors. We previously covered the administration’s confusing restrictions on American AI.
Today, POLITICO issued a lengthy critique of dysfunction at the Bureau of Industry and Security, which is responsible for controlling the flow of advanced AI technology abroad. Delays in license approvals have doubled since last year, the blacklist of foreign firms that the bureau maintains hasn’t been updated in months, and former officials suspect that thousands of high-end chips have slipped through the widening cracks.
Lawmakers’ attention might be better spent codifying and formalizing transparency and evaluation requirements, and investigating the failures in our enforcement of existing export controls.
Today’s data
Claude Code secretly tracked user data
For as long as I’ve been using AI, or even the internet, I’ve assumed that people and companies could pinpoint my location and identity if they were really trying. There are myriad ways to identify me, from tracing my IP address to using AI to recognize my writing style.
It turns out this was a little bit pessimistic. AI companies do keep records of accounts and IP addresses that access their systems, and records of the conversations those accounts have with company AI, but they don’t generally take fine-grained identification data from a user’s own system. China recently accused Anthropic of breaking this unwritten rule by quietly storing GPS and other identifying information belonging to users of Claude Code.
This was most likely an attempt by Anthropic to catch illicit use of its AI, like the mass cyber campaign it caught last year, or the “distillation” of Claude’s capabilities into cheaper AIs, or the Chinese black market in cheap access to Anthropic’s models. It’s an understandable move, and not exactly the “serious threat” or “back-door vulnerability” that China would presumably like its citizens to believe. But as far as I can tell, it was unannounced (not mentioned in the changelog) and done in a plainly sneaky fashion.
In a move that sure looks like an attempt to obfuscate the data transfer, location data was encoded in tiny details in the system prompt (the instructions sent to every instance of an AI). One of these details was the date separator (either a hyphen or a forward slash) and the other was the exact kind of apostrophe in the phrase “Today’s date is...”
It should not be news to us that AI companies can, and sometimes do, secretly modify user requests, or store user data. Remember: they keep all your conversations. It says so in the privacy policy! Chat transcripts can even be used as evidence in court.
Some sources claim the hidden tracking has since been removed. Users still expressed concern over the subterfuge, and I don’t blame them. This incident looks to me like Anthropic is flying by the seat of its pants regarding security, and it doesn’t leave me very confident in their ability to protect the data they store.
In many ways, privacy trades off against catching illegal and unethical behavior. I don’t pretend to know exactly where the line should be. But I do know this: if you were previously under the impression that AI companies are being open and responsible about their use of your data, today`s a good day to reconsider.
Dispatch from Donald
65% of Americans are concerned about AI
Results from the tracking poll “Americans on AI”
The Hill’s Miranda Nazzaro reports on a new poll tracking U.S. public opinion on AI. The poll, which is called “Americans on AI,” is run by the nonprofit Athena Insights. A new poll will be held every two weeks for the next year in order to track changes in opinion over time. Two polls have been taken so far; the third update is due on July 15.
In the most recent poll, out of roughly 1,800 participants, 65% said they were either somewhat concerned or very concerned about “AI’s growing role in society.” Only 24% said they were “somewhat excited” or “very excited.” (The percentages don’t add up to 100% because participants could answer “none of these are close to how I feel” or decline to answer.) Public sentiment was remarkably similar across party lines: 66% of Democrats and 65% of Republicans said that they felt concerned.
StopWatch will probably revisit Americans on AI in the future, but if you want more now, you’re in luck. Their data is publicly accessible at their website, where they include many more questions, give context for each one, and break down the results by party, age, gender, and race. Trendlines are also included, but with only two polling waves to track so far, they’re more noise than signal at this point.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.