Among all the debates raging about the potential downsides of AI, there’s one concern that’s bothering Silicon Valley’s AI enthusiasts the most. Their concern is that the big AI labs that sell their own models are somehow acting like Trojan horses.
The concern is that as startups and companies use Labs’ AI models like OpenAI and Anthropic, the Labs will increasingly have access to those companies’ most sensitive business information. Model makers can use that knowledge for themselves and potentially become competitors for their own customers. Those issuing such warnings range from venture capitalists like Jason Calacanis to Palantir CEO Alex Karp.
Now, in a surprising blog post published on Sunday, Microsoft CEO Satya Nadella also joined the crowd. Nadella warns that AI users, or what he calls “purchasers,” are paying twice. They knowingly spend money for the use of AI tokens, but also unwittingly hand over valuable data in the process.
“You essentially pay for intelligence twice: once in money, and once for something even more valuable: the unique knowledge you need to uncover to make that intelligence useful. The better you want your model to perform, the more knowledge you need to feed it,” he writes.
The greatest danger, he argues, is that companies are literally teaching models about the nuances of their businesses.
“Models learn from the prompts people write, the tools agents use, and especially the corrections people make when the model is wrong. All corrections are distilled into organizational know-how,” he writes.
This is “the kind of knowledge that competitors could never buy,” and yet companies are handing it over.
Nadella argues that if AI companies are free to harvest the internet to train their models, it stands to reason that in return they will be able to study, or “distill”, those models. “Distillation” is the practice of using the output of the model itself to learn how the model works and then training a new (often cheaper) model based on those insights. In February, Anthropic accused the Chinese open source model of sending Claude millions of prompts as a way to improve its own model and urged the U.S. government to crack down on export controls.
Nadella’s point is that model makers can’t have it both ways. It’s hypocritical for them to freely train on data from all over the world while restricting others from doing the same with their models.
“We need great innovation from model providers who have fair use rights to train models on publicly available data, but I find it ironic that the status quo has reversed to impose restrictive conditions on distillation,” Nadella wrote.
Nadella is particularly concerned that model makers “reserve the right to learn from customer usage and interaction data.”
Nadella’s solution sounds like something a CEO of a large cloud provider would propose. He wants companies to “retain ownership” of their data, including prompts, feedback, etc., and therefore encourages them to build their own “unique learning environments” in the cloud (which could conveniently mean Microsoft’s cloud, Azure, where the data is probably already stored anyway). He also wants companies to build what he calls an “orchestration layer.” This is essentially a way to easily switch between AI models from different providers, rather than locking in to one. Tools like AI “gateways” that allow businesses to do just this are becoming increasingly popular.
Nadella never uses the term “open source” as a way to retain ownership, but this is an obvious subtext. But there’s also another subtext.
Although many large enterprises still have some of their own data centers in addition to using the cloud, they have already begun to move to an open source model located on their own facilities (or “on-premises” in industry parlance). Idit Levin, founder and CEO of Solo.io, which develops networking and security software that helps enterprises manage AI systems, says she’s seeing this very change unfolding for her customers. After experimenting with a proprietary model maker, they start asking themselves questions. “Can we take an open source model and run it on-premises? It would do almost 90% of what the big models do, and it would be much cheaper,” she told TechCrunch. “They understand it and can control it.”
Solo.io’s technology was selected last year to power the Linux Foundation’s Agentgateway project. Her company counts companies like T-Mobile, ADP, and SAP among its customers. She sees companies increasingly adopting an on-premises open source model as the next big wave in enterprise AI usage.
she is not alone. Both Vercel (best known as a platform for building and hosting websites and recently added an AI model switching tool) and OpenRouter (a company that helps developers route requests between different AI models) are seeing a surge in traffic to their open source models. In fact, the open model accounted for 29% of all traffic routed through Vercel’s gateways last month.
This trend is likely to continue to grow, as the CEO of Microsoft, an investor in both OpenAI and Anthropic, is now publicly calling on companies to be wary of using proprietary models. “By consuming intelligence, you are creating intelligence, and what you create should be yours,” Nadella writes.
If you buy through links in our articles, we may earn a small commission. This does not affect editorial independence.
