Coherence in this context means the ability of a GPU to talk to other GPUs in a computer GPU cluster. The expert consensus was that the limit was around 25 to 30K GPUs. Elon Musk found a way to extend that limit to possibly one million GPUs. Considering that the human brain has some hundred trillion synapses (connections) there is still room to grow.
Elon and xAI BREAK the Coherence Bottleneck–with HUGE Consequences!
These breakthrough sometimes happen when we remember some trivia buried deep in our memory. I had such a moment back in 1986 or 87 working on our Desktop Help, a Mac Desk Accessory. Computer memories were tiny by today’s standards and Mac Desk accessories had a limit set by hardware, the Motorola 680x0. By 1986 very few programmers had any experience with machine code, they relied on high level languages like B, C, and Pascal. My first programming experience was very close to machine language. Looking at Mac’s addressing technology I found a way to double the memory available to Mac Desk accessories without really breaking Apple guidelines. The Address table used one bit as the plus-minus sign. We didn’t need the sign and by adding one bit to the address table we doubled the accessible memory. Thinking in machine language terms the solution is trivial but not available to programmers using high level languages.
One has to wonder how much trivia Elon’s brain has accumulated that he puts to good use.
YES!
And this fact, as it is fact not just metaphor that all problems have levels of granularity, with machine language in computation being the fine grain and each level up from that extending generality and power paid for by losing some control and “vision” [best word I can come up with on only one cup of coffee when I am sick as a drunk old dog, but captain or someone of you can probably propose a better].
Long ago far away I worked on a team designing advanced, fast, very low power level military avionics, and that was when I discovered
and that it allowed me to easily dive down from “compiler” level context to “machine” level context. For those problems it was like being a wizard with direct communications with the “god level”. Like Captain finding that crucial extra bit.
The collapse of stable public knowedge in the face of “like based” social media strikes me as a telling instance of the problem of granularity in knowledge.
Structural integrity might be a better term. Writing code is easy, testing/debugging code is hard, maintaining code is near impossible. FORTH is one of the most powerful languages ever invented and almost impossible to maintain by new coders. The code is compact and fast but every time you write code in FORTH (and LOGO) you are inventing new words creating new languages that no one else speaks and which you yourself forgets. The curious thing is that LOGO, having the same structure, is less of a problem because the words describe the images the turtle is drawing giving it better structural integrity.
The turtle has a pen attached to its tail. If the tail is down and the turtle moving, it draws a line. Otherwise it does not. Those MIT scientists are brilliant communicators!
After six decades writing code in a multitude of languages I found the best one for creating web apps to be Object Oriented php even while I hate Javascript and don’t particularly like CSS. With the help of Google and some great free online tools it works great. But you can’t just start writing code, you need to create the architecture before you do the engineering. Structural integrity is key. If you don’t have it the code becomes spaghettified.
What is spaghettification?
In astrophysics, spaghettification is the tidal effect caused by strong gravitational fields. When falling towards a black hole, for example, an object is stretched in the direction of the black hole (and compressed perpendicular to it as it falls). In effect, the object can be distorted into a long, thin version of its undistorted shape, as though being stretched like spaghetti.
In computer programing spaghettification is pages of endless code impossible to unravel. Object Oriented Programming, omitting all the complication experts have piled on it, is excellent to keep things simple and orderly. I took my inspiration from HTML5’s Semantic Elements:
What are Semantic Elements?
A semantic element clearly describes its meaning to both the browser and the developer.
Structural integrity is dead on to my target but misses the element of visibility and clarity I wanted and then experienced when I went Forth-ing.
I never learned Logo (nor its daughter Smalltalk), although I was strongly drawn to both. By that time I was focused on making as much money as fast as possible, and so I did maintenance work on Fortran (yuck!!!) spaghetti code for Quotron (really nasty stuff),
also writing a lot of the code for their Pacific Stock Exchange options floor implementation.
After I had enough $$ I ran off and did my gay rights and other political stuff.
An interesting observation that might have more to do with the coder than with the code. For me software has two domains, structure and flow. I can visualize structure quite readily. I have a much harder time with time dependent flow which became even more complicated with the internet because you have two or more computers interacting over time and space. For me structural integrity implies visibility but it might not for others who have a different approach to coding.
The title “Object Oriented Programing” (OOP) implies structure but the founder (or one of the founders) stated that OOP was about messaging. Same dichotomy, structure and flow.
Thanks Captain, that explanation was very clear and useful.
Coding as we knew it (as it was being invented!) might become a subfield not only of linguistics and management but also of anthropology….and psychology.
Well, here’s an anthropologist who stopped counting languages used for development when I got to 50. I spent the last part of my career at the opposite end of the spectrum from assembler … although my first published paper was based on assembler … working with a 4GL from Progress Software. I had developed a tool called Specification Driven Development which allowed a programmer to create the bulk of an accounting application at 1000 lines of 4GL ready for integration testing per hour (3X to 10X that for a 3GL). And then I was working on enhancing that to drive the whole thing from UML.
It has been a long time, but as I remember something north of 2000. A surprising number of which are spoken on the island of New Guinea because there is a steep mountain range down the middle and steep walled valleys up the side so that there are often only 3 to 5 little villages in a “tribe” that speak a unique language - a few thousand people at most, all of which are related. A culture with a strong emphasis on warfare helps keep the separation. Disclaimer, these comments are based on 50 year old memories.
