Andrej Karpathy: Software Is Changing (Again)

TL;DR

Software is undergoing a paradigm shift, evolving from explicit code (Software 1.0) through trained neural network weights (Software 2.0) to LLMs programmable via natural language prompts (Software 3.0). LLMs are fallible, human-like simulators with cognitive deficits (hallucination, anterograde amnesia, gullibility) but superhuman knowledge.

Key insights

• Software has changed fundamentally twice in the last few years after remaining largely unchanged for 70 years; a huge amount of existing software must be written and rewritten.
• Software 1.0 is explicit code (e.g., Python, C), Software 2.0 is neural network weights produced by tuning datasets and running optimizers, and Software 3.0 is prompts in English that program an LLM.
• At Tesla, the autopilot's Software 2.0 stack grew while its C code (Software 1.0) was deleted as capabilities were migrated into neural networks.