The most dramatic optimization to nanoGPT so far 25% speedup is to simply increase vocab size from 50257 to 50304 nearest multiple of 64. This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2.
Andrej Karpathy quote
Advertisement
More from Andrej Karpathy
-
Earth as a dynamical system is a really bad computer. A lot of information processing is concentrated in a few tiny compute nodes brains, chips with terrible interconnects, even as bad as use of physical translation and air pressure waves. And powered primitively by combustion.
-
A 4-year-old child actually has a few hundred million years of experience, not 4. Their rapid learning and generalization is much less shocking magical considering this fact.
-
It would be best if people made strong statements that are understood to be only 90% true, and ignore the counterexample police. This saves time and makes direction of statements clear.
-
For us it's much more becoming a story of How do humans and computers collaborate to actually create these vector space data sets?
-
If you vibrate the electromagnetic field just right, cars passively awash in the radiation for a while will suddenly drive better.
-
It's been a while since I last played Factorio, but I can't stop thinking about it, and of the entire economy as being an MMORPG version of it. Factory Town, RimWorld, Banished, Dawn of Man etc are all great too but somehow pack less long-term punch.