Karpathy's Blog: Pure Python AI From Scratch
Andrej Karpathy distills neural nets into minimal Python code—200 lines for GPT training/inference—plus RL, RNNs, and human baselines on vision tasks.
Minimalist AI Implementations in Pure Python
Build GPT from scratch in 200 lines of dependency-free Python for training and inference, proving core LLM capabilities need no frameworks. Recreate LeCun et al.'s 1989 backprop neural net—the earliest real-world end-to-end example—using 33 years of deep learning progress to benchmark historical vs. modern methods. Train character-level RNNs to generate poetry, LaTeX math, and code, revealing their unreasonable effectiveness for sequence modeling. Implement deep RL to play Atari Pong from raw pixels via policy gradients, weighing pros/cons like sample inefficiency. Classify 2 million scraped selfies as good/bad with CNNs, visualizing what networks 'think' about images. Fool ImageNet linear classifiers with imperceptible perturbations, showing even simple models are brittle beyond ConvNets.
Human Baselines and AI Progress Benchmarks
Humans achieve better than 6.7% top-5 error on ILSVRC 2014 ImageNet vs. top ConvNets, but manual CIFAR-10 labeling exposes dataset ambiguities driving DL gains. In 2012, computer vision lagged far behind human performance, underscoring AI's distance from general intelligence. Project 33 years forward: today's DL will seem primitive by 2055, just as 1989 nets do now.
Productivity Hacks and Training Recipes
Quantify daily productivity by tracking active windows and keystrokes on Ubuntu/OSX, generating HTML visualizations for insights (code on GitHub). Train neural nets effectively with a recipe: practical steps like batch norm, learning rate tuning, and gradient clipping to hit strong results reliably. Scrape Hacker News front page every minute for 50 days to model story rise/fall dynamics and success factors. Build Chrome extensions in few JS lines for Twitter auto-refresh and rare-tweeter highlights, as a survival skill for devs. Switch blogs from WordPress to Jekyll for static speed and control.