OT: Some AI Observations

[brittlerock]

As some of you know, and the rest of you who read this post are about to find out, aside from being an investor I’m a musician.

Huh? So what does have to do with investing, much less, AI. Let me explain. As a musician I watch the YouTube videos made by Rick Beato. Rick is highly experienced in many aspects of the music business and he’s an educator. I find his videos interesting, informative and often thought provoking.

As you might imagine, folks from almost every aspect of the music business are concerned about the potential (mostly negative) impact of AI on a wide range of music business related jobs. Songwriters, performers, producers, arrangers, engineers - you name it, have all expressed worry about how AI might take over their jobs.

Which brings me to the meat of this post. I’ve read several threads on this board which have some discussion about using AI as a source for investing information. Almost all of these threads contain some cautionary words about the possibilty of receiving misinformation.

Rick recently released a video wherein he explores the possibility for getting bad information from AI, his observations are not strictly limited to subjects related to music. I think you may find the cited video very interesting.

At present, I’m in China. China erected a firewall that blocks access to many sites, among them, all Google related sites. I have a VPN which works some of the time on my Android phone and almost none of the time on my Windows pc. In other words, I can’t (don’t know how) to provide a link, but don’t dispair. Search for “I fried ChatGPT with one simple question.”

[Smorgasbord1]

I think it’s wrong to look at the general-purpose, publicly-available versions of AI out there and conclude that it’s not useful, or not even going to be useful.

For instance:

A new generative AI approach to predicting chemical reactions

Many attempts have been made to harness the power of new artificial intelligence and large language models (LLMs) to try to predict the outcomes of new chemical reactions. These have had limited success, in part because until now they have not been grounded in an understanding of fundamental physical principles, such as the laws of conservation of mass. Now, a team of researchers at MIT has come up with a way of incorporating these physical constraints on a reaction prediction model, and thus greatly improving the accuracy and reliability of its outputs.

and

In their comparisons with existing reaction prediction systems, Coley says, “using the architecture choices that we’ve made, we get this massive increase in validity and conservation, and we get a matching or a little bit better accuracy in terms of performance.”

We are in AI’s early innings. Transport yourself back 120 years and try to decide about the future of the automobile from people attempting cross-country trips back then. The first successful trip took over 2 months. Muddy roads, no fuel infrastructure, mechanical brakes, only 20 HP, etc. That was NOT a harbinger of what was to come. Similarly, today’s AI struggles are, I believe, also not a harbinger of what is to come.

[brittlerock]

@Smorgasbord1 My take away from Rick’s experiment wasn’t the conclusion that AI isn’t useful. In fact, you might recall that he discussed an application of AI with an example of how he has used AI to generate songs - not very good songs, but songs none the less. He used a few different AI tools to create the different aspects of songwriting; lyrics, melody, groove, etc. and then using the separate outputs as the input for another AI tool to generate a demo.

My take away was that AI is severely limited by the availability of training data. When it comes to the tasks of producing a song, recording a song, mixing a song at a professional level, AI hasn’t the ability to execute because of the dearth of training data. There’s some information on these topics, but precious little documentation from the experts in the field, hence AI can at best provide generalities about how to accomplish these tasks.

Separately, there’s a ton of training data on the internet when it comes to songs. There was 100 million songs on Spotify at some point in 2024. About 60,000 new songs get uploaded daily. And Spotify has undertaken the task of assembling some of those songs that meet certain quality/popularity criteria into playlists of different genres.

I think he demonstrated his point pretty well at the start of the video when he asked ChatGPT to answer the question, what is 52!. After ChatGPT failed, he then used an app I didn’t know existed to query all the publicly available AI models with the same question where he got varying answers at different cost from each tool.

Those that had used YouTube for training data came up with the right answer quickly due to the fact that this question had already been answered in a YouTube video that addressed the question of how many different ways might the cards of standard card deck get arranged by thoroughly shuffling them. Apparently, his gifted kid had also watched that particular video and was able to recite the astonishingly large number from memory, including the names of all the units above a trillion.

I think the example that you cited regarding the prediction of possible results of complex chemical reactions clearly reenforces Rick’s conclusion rather than refutes it.

The available AI tools failed this task until the researchers at MIT were able to determine what training data was absent. After the AI was provided the appropriate training data, the tool performed admirably. Rick’s video clearly demonstrated that until the training data deficiencies are addressed, AI will never perform well at certain tasks. In the case of certain aspects of creating a professional level song, there is virtually no training data available. If there were, Rick would know about it.

Really, there’s nothing new about this observation. I don’t remember when I first heard the word “gigo,” but it was a very long time ago. Despite all the potential AI presents, gigo is still true.