Diffusion is a training framework—not architecture—that creates extra samples by gradually noising clean data over 1,000 steps, outperforming autoregressives on 25-100M tokens where data is limited but compute abundant; lags in text due to slow inference and infrastructure.
Train GPT, RNNs, RL Pong, and Bitcoin tx in pure Python with zero dependencies—distilling neural nets to essentials in under 200 lines.
Sander from DeepMind reveals data curation trumps model tweaks, latent autoencoders enable scale, diffusion denoises via spectral autoregression for superior audiovisual generation.
PCL algorithm integrates predictive confidence scores into LLM RL rewards via ensembles and blended token/sequence signals, enabling adaptation to nonstationary changes without retraining.
Transformers ended 30 years of sequential processing flaws by using self-attention, where every word weighs relevance from the entire sentence context, powering GPT and all modern LLMs.
Efficient LLM inference hinges on mmap for lazy memory loading (e.g., <10s startup on llama.cpp) and quantization like GGUF K-Quants or AWQ/EXL2 to shrink 15GB models while preserving quality via salient weights and mixed precision.
Andrej Karpathy distills neural nets into minimal Python code—200 lines for GPT training/inference—plus RL, RNNs, and human baselines on vision tasks.
Raw CIFAR-10 pixels yield 65% test accuracy; normalization/standardization lift to 69%; geometric augmentation maintains ~67%; photometric brightness/contrast crashes to 20%; combined pipeline with deeper CNN hits 87%.
DeepMind pushes Gemini beyond LLMs with omnimodal embeddings for unified retrieval, weather models beating physics sims (GraphCast: 15-day forecasts; GenCast: 97% benchmark accuracy), and Genie world simulators for interactive 3D environments.
Sebastian Raschka's gallery visualizes 70+ LLM architectures with diagrams, key specs like KV cache costs, attention types, and a diff tool—ideal for comparing dense vs. MoE designs and inference tradeoffs.
QK^T in attention computes kernel similarities between queries and keys; Mercer's theorem proves it's a valid positive semi-definite kernel, making softmax a mathematical necessity for normalization, not just architecture.
Train small 'student' models on large 'teacher' models' soft probabilities—not just labels—to match performance while slashing size, speed, and costs by 53x by 2025.
Parcae enforces looped transformer stability via negative diagonal matrices in a dynamical system, outperforming baselines and achieving 87.5% of a twice-sized Transformer's quality at half parameters.
Step-by-step Colab guide: generate 2D Darcy datasets via GRF & finite differences, implement/train FNO operators and PINNs, add CNN baselines, benchmark inference speeds for fast physics surrogates.
Word2Vec learns dense word vectors by predicting local contexts with CBOW or Skip-gram, clustering similar words like 'cat' and 'dog' via repeated gradient updates from shared neighborhoods.
Fuse LSTM operations into nngraph module to batch 4 GEMMs, slashing overhead vs standard nn.LSTM (optimized by @jcjohnson).
Custom Torch nn.Module normalizes each row of n x d input tensor to unit L2 norm, with efficient batched forward/backward passes for training.
Build a vanilla RNN language model from scratch in ~170 lines of NumPy: processes text chunks of 25 chars, trains with BPTT and Adagrad, generates samples after 100 iterations.
Efficient pure NumPy LSTM processes batched sequences (n,b,input_size); init with Xavier + forget bias=3; verified via sequential match and numerical gradients.
Train a 2-layer NN to play Atari Pong from raw pixels using REINFORCE policy gradients. Uses 80x80 binary diff frames, discounts rewards with gamma=0.99, standardizes advantages, RMSProp updates every 10 episodes. Converges on CPU in hours.
Andrej Karpathy distills neural nets, LLMs, RL, and Bitcoin into 200-500 line pure Python scripts—no deps needed—to teach core mechanics hands-on.
AI generates undetectable fakes that exploit human trust shortcuts—train yourself to pause and question realistic audio, video, or text instead of believing instantly.
TurboQuant rotates KV vectors before quantizing to 3.5 bits/channel (quality-neutral) or 2.5 bits (minor degradation), plus error repair, yielding 6x memory savings and up to 8x speedups for long-context LLMs.
Roofline analysis shows batching 2000+ tokens amortizes weight memory fetches, slashing per-token cost 1000x; fast modes use tiny batches for low latency at 6x price.
DeepSeek-V3, a 671B param MoE LLM (37B active per token), trained on 14.8T tokens using FP8 and optimized infra for 2.8M H800 GPU hours ($5.6M total), outperforms open-source models and rivals GPT-4o/Claude-3.5-Sonnet in code, math, and reasoning.
Catalog of 70+ Microsoft papers on 1.58-bit LLMs for CPU inference, zero-shot TTS, long-context scaling to 1B tokens, and agentic reasoning via distillation and sparsity.
PhysicsNeMo equips developers with an open-source PyTorch-based toolkit to build, train, and fine-tune deep learning models incorporating physics constraints, supporting 20+ pre-implemented architectures for weather, mechanics, and more.
World models train on experience streams to predict cause-and-effect dynamics, creating compact internal simulations for efficient planning and physics understanding—surpassing LLMs' token prediction.
