Fix Randomness First for Stable ML Pipelines
ML systems fail from unstable pipelines, not bad models—control randomness by setting seeds across random, NumPy, and PyTorch to ensure reproducible results.
Pipelines, Not Models, Break ML Systems
After 4+ years building ML systems, the core failure mode isn't weak models but unstable pipelines that produce inconsistent results. A one-time success turns into quiet failures without disciplined stability practices. Treat stability as a non-negotiable discipline, not an afterthought.
Enforce Reproducibility by Seeding Everything
Randomness turns models into unreliable slot machines—results vary per run, undermining debugging and deployment. Fix it with a global seed function covering all sources:
import random
import numpy as np
import torch
def set_seed(seed=42):
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
set_seed(42)
Call this early. Key caveat: Seeds don't fully eliminate non-determinism in some GPU operations—explicitly configure those for true reproducibility.
Note: Article outlines 9 rules total but details only the first here.