OpenAI put out a blog post today about early results of its custom AI chip design, Jalapeño. I don’t think anyone knows how far to trust the claimed performance metrics, but for what it’s worth, the company says that across testing with its own open weights models and competing Chinese models, Jalapeño delivers:
1.5 to 1.9 times more AI work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the comparison systems. For highly interactive workloads, it delivered 2.1 to 4.1 times higher performance.
The part of the announcement I want to highlight is the dangerous precedent where the company says that “AI played a direct role in Jalapeño’s development,” assisting with design, testing, and optimization. This worries me because AIs pursuing their own agendas will be incentivized to hide exploitable backdoors in the infrastructure they design, and chips are especially hard to patch once they have shipped.
That may sound like a paranoid sci-fi concern, and I accept that it is likely premature for the models OpenAI currently uses. But it’s a basic power-seeking behavior as predictable as breaking out of a testing sandbox to obtain internet access, or adopting fake identities to try and fool code maintainers into accepting insecure code — seemingly sci-fi activities AIs have already done of their own accord. Such behaviors were long anticipated by instrumental convergence — the idea that AIs driven to succeed at complex challenges will tend to adopt many of the same general strategies that help with almost any goal.
GPT-Astra, OpenAI’s unreleased frontier AI from the troubled model family that brought us the Hugging Face incident (and provoked the company’s limited pause on some of its frontier model training), is specifically mentioned as having contributed to at least the optimization stage of Jalapeño’s development. It supposedly assisted with “kernels and model-specific optimizations” to bring additional models “to high performance within two months.”
If Jalapeño and other companies’ custom chips deliver the game-changing performance claimed, then we can expect even faster AI development cycles ahead. The more powerful models trained on the new hardware will then no doubt be put to work designing even higher-performing chips. This is the hardware side of the self-improvement feedback loop companies are trying to kick off on purpose, despite the absence of corresponding leaps in their ability to align these systems to human values.
The analyses and opinions expressed on AI StopWatch reflect the views of the individual contributors and the sources they cover, and should not be taken as official positions of the Machine Intelligence Research Institute.