Prioritizing Actionable Metrics Over Dashboard Green-Lights

At scale, standard monitoring often fails to capture the reality of GPU health. Gabriel Menezes emphasizes that 'GPU utilization' is a misleading metric; instead, Krea.ai tracks tensor core utilization as a more accurate proxy for actual compute efficiency.

Infrastructure failures at scale are often silent, manifesting as communication timeouts rather than clear error logs. The team's strategy is to stop treating individual node crashes as mysteries. If a node fails, they restart the job; if it persists, they replace the hardware. Key operational rules include:

  • Thermal Thresholds: Any GPU exceeding 78°C is immediately pulled from the cluster. Attempting to debug hot cards only destabilizes the entire training run.
  • Custom Fabric Metrics: Off-the-shelf tools often miss InfiniBand and NVLink errors. The team built custom collectors to monitor packet loss and wait times on the fabric, which proved critical for identifying cross-node communication bottlenecks.
  • Aggressive Checkpointing: To mitigate the impact of inevitable crashes, the team uses a high-performance filesystem capable of writing a terabyte of data in under 30 seconds, allowing for frequent (20-30 minute) checkpoints without stalling training progress.

Fluid Resource Orchestration

To maximize hardware investment, Krea.ai runs training and production inference on the same cluster using a priority-based scheduling system. The goal is to ensure researchers never have to worry about GPU availability while maintaining production uptime.

  • Gang Scheduling: Using the Kueue project, the team implements two-tier priority. Training jobs are assigned high priority; when they launch, they automatically preempt production inference tasks.
  • Virtual Kubelet for Inference: When training jobs occupy the cluster's GPUs, production inference is seamlessly migrated to external providers via a 'fake' Kubernetes node. This is managed by a custom implementation of Virtual Kubelet.
  • Automated Migration: To prevent production downtime, the system uses Kubernetes taints and tolerations. When GPUs are available, a taint is added to the node; a descheduler then slowly migrates inference tasks back from external providers to the internal cluster. This gradual migration ensures that traffic is never dropped, avoiding the 'all-at-once' disruption that would occur with a 'NoExecute' taint.