To make the analogy explicit, in Software 1.0, human-engineered source code e.g. some .cpp files is compiled into a binary that does useful work. In Software 2.0 most often the source code comprises 1 the dataset that defines the desirable behavior and 2 the neural net architecture that gives the rough skeleton of the code, but with many details the weights to be filled in. The process of training the neural network compiles the dataset into the binary — the final neural network. In most practical applications today, the neural net architectures and the training systems are increasingly standardized into a commodity, so most of the active software development takes the form of curating, growing, massaging and cleaning labeled datasets.
Andrej Karpathy quote
Advertisement
More from Andrej Karpathy
-
Earth as a dynamical system is a really bad computer. A lot of information processing is concentrated in a few tiny compute nodes brains, chips with terrible interconnects, even as bad as use of physical translation and air pressure waves. And powered primitively by combustion.
-
A 4-year-old child actually has a few hundred million years of experience, not 4. Their rapid learning and generalization is much less shocking magical considering this fact.
-
It would be best if people made strong statements that are understood to be only 90% true, and ignore the counterexample police. This saves time and makes direction of statements clear.
-
For us it's much more becoming a story of How do humans and computers collaborate to actually create these vector space data sets?
-
If you vibrate the electromagnetic field just right, cars passively awash in the radiation for a while will suddenly drive better.
-
It's been a while since I last played Factorio, but I can't stop thinking about it, and of the entire economy as being an MMORPG version of it. Factory Town, RimWorld, Banished, Dawn of Man etc are all great too but somehow pack less long-term punch.