Datadog Integrates with OpenAI ChatGPT to Help Organizations Monitor AI Usage, Costs and Performance
I don’t really understand what it’s about but it seems to answer those who felt that Datadog wasn’t catching on about AI.
Saul
PS I don’t have a position in DDOG.
My understanding is that they released a native integration for apps that uses OpenAI’s ChatGPT feature: you send text to OpenAI, pay by the length of text, and get the result back. Datadog lets you insert a line of code to track how much of each of the above is happening.
OpenAI’s ChatGPT service for commercial use charges by length of text. The service is quite expensive by normal API standards (incredible pricing power from OpenAI!) As a commercial user, you can either 1) write your own code to figure out how much you are sending and calculate the total yourself - this usually involves writing bespoke code to dump the metrics from you app to some data warehouse like Snowflake, then have analytics people set up a dashboard — a multi-team effort — or 2) you go the easy route and call the line of code from Datadog then set up the dashboard in the Datadog page yourself.
Integration is how Datadog help developers build dashboards to track and monitor metrics that are relevant to their applications. I’ve seen people described that Datadog has “native integration to every last SaaS service you have never heard of” when compared to their competitors. It is not surprising that they will release one for OpenAI because this is their bread and butter. They don’t sell “AI” but the plumbing that goes into it.
Well, not quite, at least for me. This is DataDog monitoring one’s usage of the AI service known as ChatGPT. It’s not DataDog using AI itself to improve its monitoring abilities.
I did some first-level internet research and came across these links:
DataDog WatchDog
DataDog WatchDog AI
DataDog’s Machine Learning
Watchdog is an algorithmic feature for APM performance, infrastructure metrics, and logs that automatically detects potential application and infrastructure issues. It leverages the same seasonal algorithms that power anomalies and dashboards.
Watchdog analyzes billions of events and learns what “normal” behavior looks like in order to proactively provide insight to users for anomalies they didn’t anticipate. The two new capabilities of Watchdog take this one step further.
Essentially, what DataDog is doing here is setting up programmatic algorithms, and then watching what values the parameters have over some period of time. Then, if the parameter values exceed those, you get notified.
Anomaly detection addresses one of the core challenges in monitoring dynamic, responsive, ever-scaling infrastructure: How to define normal versus abnormal performance. Setting static thresholds often leads to false alarms due to normal variations in key metrics like website traffic and customer checkouts, which tend to rise and fall depending on the time of day, day of the week, or day of the month. Anomaly detection accounts for those expected variations, as well as long-term trends, to intelligently flag behavior that is truly unexpected. Datadog’s anomaly detection algorithms are rooted in established statistical models, but have been heavily adapted for the domain of high-scale infrastructure and application monitoring.
Perhaps a good analogy would be home monitoring. It watches your house for a while, keeping track of temperature extremes, rate of rise, and let’s say window and door opening and closing for when, how long they stay open, how many are open at any time, etc. Then after a period of time you turn that around, and the system then looks for temps above/below the extremes or rising faster than previously, or windows staying open longer, etc.
That’s a great feature, and sure it’s a machine that’s learning, but it’s not the kind of generative, large model, neural net AI that is all the rage right now. It still needs algorithmic programming, and needs to understand seasonal loads - for the house that might be indoor temps in winter vs summer, for eCommerce that might be pre-Xmas loads versus mid-year loads.
That said, I wouldn’t expect DataDog to be talking much about potential new features they don’t have yet. OTOH, I would have expected the CEO when addressing the impact of AI to have talked about more than just AI creating more apps and DataDog monitoring AI workloads.
That said, holding some DDOG, I’m happy with what I see as the market over-reaction to the press release.
I should add that my AI concerns are probably way early - I often get the timing wrong on technology adoption and impact.
Saul –
they aren’t actually incorporating ChatGPT nor adding an AI features.
Datadog is instead building a system for customers using Open AI’s models (like ChatGPT and DALL-E) to track and monitor their usage and costs of calling into the API engines.
it’s a good idea to quickly provide a service over the coming wave of AI use… but i expect a lot more to come as Datadoog actually incorporates more AI use over the pool of observability data it holds.
-muji
ps the Watchdog you dug up Smorg is their long time AI/ML engine over cust data. that has been in place a long time, since mid-2018, and drives all the event coorelation and issue alerts in Datadog.
what I think Saul was hoping for, and what I think will ultimately appear, is something like Microsoft and Google are building for security. A separate engine that uses one of these LLMs to build a knowledge base over a customer’s IT and app stacks in a secure/private way, so that DevSecOps users can ask questions of it and trigger actions.
All us DDOG investors hope for that and more. Is there any evidence to support that, especially since the “limited AI” WatchDog AI blog page I linked is dated just last year?
last year’s news was 2 new ML-backed features, great features but nothing related to this rise of LLMs.
our only evidence is the last bit from the CEO on the very last Q&A in the earnings calll:
Obviously, the expectation for some of those products are changing over time too. You know that everyone can see what can be done with AI. We really expect to see a lot more of that. So, I guess we’ll share more on that in the near future.
well… that and if they DON’T adopt LLM capabilities I described above, they lose their competitive edge.
Hi Muji,
When you suggest in the above, that ‘DevSecOps users will need a secure private way to utilize an LLM integrated into a customers IT and AppStack’, did you leave out the Data Lake? And if so, doesn’t that almost require that a Datadog like solution would need to be built natively in a Platform like what Snowflake offers?
I just think that that including the Data Lake or Lakehouse is a need to have not just a want to have. What do you think?
Best
Jason
jason, i think you are mixing the use cases.
absolutely a data lakehouse is useful, as a place to capture all operational data about the business. this can be leveraged by LLMs to build a queryable knowledge base over that data, and to extract meaning out of unstructured data like PDFs, images, and videos.
what i am suggesting with observability platforms like Datadog is that an LLM could be used to build that same type of queryable knowledge base over all IT infra and app stacks utilized. this is a different data set – though I admit its confusing as data lakes can be used to store and archive IT op data as well as business data (as well as security and any other form of enterprise data).
think of the difference as using AI over BI vs ITOps needs by an audience of business users vs DevSecOps. tho these different data sets could certainly intermix - a combo that may be of value to cloud-native platforms where cloud = the entire business.
but for now, Datadog is not used for BI beyond what can be extracted from apps directly (say, e-commerce sales and eyeballs on content). it isn’t used to track sales financials and physical assets like store fronts and inventory. that is a role for operational databases and ERPs… and ultimately Data Lakehouses.
Lines are blurred, but i think it best to mentally frame data needs and AI potential in that way … DevOps vs Security vs BI.
-m
Well if DDOG ain’t goin this direction… others are.
New Relic announced Gen AI assistant over your observability data…
IMO, Datadog, New Relic and other companies in the observability field need to embrace AI and incorporate it into their product suite lest AI simply make their product suite obsolete.
OK, I admit I really don’t have an in depth understanding of these products. But, based on a very high level, skim the surface knowledge of what these tools do it seems pretty clear that AI prompts hold the potential of eliminating mountains of carefully crafted human constructed code. It boils down, I think, to the training data. I’m not sure how much extra effort needs to be invested in development of algorithms. It seems a lot of that is already inherent in the capabilities that are already public.
Please correct me if I’m wrong as I’m getting somewhat nervous about my position in DDOG.
It seems to me like we’re more likely to overrate the impact of AI than to underrate it. Long term it will change the world like other major technical innovations have (like iphones/mobile did, for instance). Short term, AI will grab headlines and perhaps exacerbate trends already in place – like muddying the waters on what’s real and what’s not real, like the internet has been doing for years.
I see it as more likely to be a boon for Datadog etc than a bane. My guess is the market does too…because I don’t see the earnings report they delivered as a reasonable cause for its 31% rise this month – it simply wasn’t that good. I’ve trimmed my Datadog a lot, so maybe that is one reason I’m not as fearful of AI as some are. But still, I do think we’re prone to overreact – and to get completely wrong the ways these things will play out.
Bear
@PaulWBryant, I completely agree on your timing thoughts. I’m usually way ahead of the curve on the impact of technology adoption - great for job hunting but bad for investing.
That said, I am going to be watching closely to see how AI impacts the Observability market. As @CMF_muji points out, other companies are already looking at using AI to enhance their product offerings. DataDog will have to do that. We don’t yet have insights into what development projects are/aren’t going on within DataDog, and there’s no real world indication that the early tools, such as the one Muji linked to from New Relic, are actually useful/better.
That said, I’ll repeat what I said earlier about expecting DataDog’s CEO to have at least talked about the potential for AI-based product offerings from DataDog. That would not have Osbourned them, as once you’ve instrumented for DataDog adopting new products is easier.