I am trying to understand why hardware companies like Nvidia and Super Micro Computer are benefitting dramatically more from AI then software companies like Cloudflare, Snowflake, and Datadog.
NVDA and SMCI are massively raising guidance, assuring visibility multiple quarters out, and producing huge increases in growth. SNOW, NET, and DDOG are effectively blind sided by what the demand environment looks like and putting up guidance that is lower then the market expected.
First we have companies like NET and SNOW saying things that would make an investor think they are huge beneficiary of AI. Here’s what NET said two quarters ago,
Our largest R2 customer, is an AI company and they’re growing at just extraordinary rates as they put more data into their models
And here is what SNOW said this past quarter,
And with AI, I mean it’s going to drive a whole other vector in terms of workload developments
Now comparing to the incredible report NVDA had, here’s some of the more interesting bits they had about massive demand for AI,
Generate AI is driving exponential growth in compute requirements
CSPs around the world are racing to deploy our flagship Hopper and Ampere architecture GPUs to meet the surge in interest from both enterprise and consumer AI applications for training and inference.
We expect this sequential growth to be largely driven by Data Center, reflecting a steep increase in demand related to generative AI and large language models.
Our generative AI large language models are driving this surge in demand, and it’s broad based across both our consumer Internet companies, our CSPs, our enterprises, and our AI startups.
And what happened is when generative AI came along, it triggered a killer app for this computing platform that’s been in preparation for some time. And so now we see ourselves in 2 simultaneous transitions.
With generative AI becoming the primary workload of most of the world’s data centers generating information, it is very clear now that – and the fact that accelerated computing is so energy efficient, that the budget of a data center will shift very dramatically towards accelerated computing, and you are seeing that now.
We’ve seen it in a lot of places now that you can’t reasonably scale out data centers with general purpose computing and that accelerated computing is the path forward. And now it’s got a killer app. It’s called generative AI.
So here you’ve got the CEO of Nvidia saying this is a iPhone moment, a killer app, driving incredible demand, and giving them visibility into the future.
Now here is an interesting paradox, the generative AI models are a software invention but they do not benefit software companies as much as hardware ones.
Here’s the crux of the issue, this is how much ChatGPT (OpenAI) spends per day: According to his analysis, running ChatGPT costs approximately $700,000 a day . That breaks down to 36 cents for each question. ChatGPT creator OpenAI currently uses Nvidia GPUs for computing power for the bot as well as its other partner brands.
Each prompt or query to ChatGPT and like products uses an absolutely MASSIVE amount of compute. It costs them 36 cents per query to run every single query. They charge $20 a month for unlimited queries on ChatGPT, so they lose a lot of money on customers who query often.
Also consider Bard by Google, it is completely free, and rivaling ChatGPT in terms of results already.
Let’s thing about what happens on a typical query for a company that is using an LLM model like ChatGPT where they are setup with R2 from Cloudflare, Datadog for logging, and Snowflake for data warehousing. Let’s also assume this same company uses chips from Nvidia and Super Micro Computing.
Datadog may log the query/prompt the user inputs, and possibly log the response that GPT output (or not). This is simply a linear increase, or a single log line per query, which doesn’t add much to growth.
Let’s look at R2/Cloudflare Workers. It’s relatively cheap, the client company is charged for hitting a database or a call to the Worker. While it’s consumption based, it’s still relatively linear in terms of growth.
On the Snowflake side the GPT query might be send over and stored for some analytics, where the business analyst uses the product to create some informational reports. Again linear style growth.
Now turning to what happened on the hardware. The GPT query reaches out and runs an EXPONENTIAL amount of increase of compute to gather results. It literally does billions of calculations on each query. This is way more demand on the GPUs and compute than anything like gaming or even cryptocurrency is sending to the data center.
Generative AI computes results from scratch every single time. So if a user goes there and types ‘Tell me some good restaurants in New York’ ten times, it doesn’t store anything. It does those billions of calculations each of those ten times, costing 36 cents per query or $3.60 total, and thus drives the exponential demand for hardware.
Meanwhile, the software gets a linear increase. An extra row or log in a database, or an extra invocation of a database call and worker.
