Verifying Hardware and Data Placement

Before optimizing, you must confirm that JAX is actually utilizing the GPU. Start by running nvidia-smi to verify hardware visibility, then use jax.devices() and jax.default_backend() to ensure JAX is targeting the GPU rather than defaulting to the CPU.

Once confirmed, treat data placement as a primary concern. Unlike NumPy, where arrays live on the host, JAX arrays reside on the device. Operations like np.asarray(y) or float(loss) pull data back to the host, forcing the GPU to wait and creating significant performance bottlenecks. Avoid these operations inside hot loops to prevent host-to-device data transfer latency.

Mastering JIT and Tracing

JAX uses the XLA compiler to transform Python code into optimized machine instructions. The jit transformation is powerful, but it relies on tracing. Because JAX traces code rather than executing it line-by-line, standard Python control flow (like if statements based on array values) will fail. Use JAX-specific primitives like jnp.where, lax.cond, and lax.scan to maintain compatibility with the tracer.

Furthermore, compilation is tied to input shapes and data types. If your input shapes change (e.g., ragged batches or variable sequence lengths), JAX will trigger a new compilation, which is expensive. To maintain performance, use padding and masking to keep input shapes stable, allowing JAX to reuse existing executables.

Profiling Asynchronous Execution

JAX dispatches GPU work asynchronously. A common mistake is measuring the time it takes for Python to request work rather than the time the GPU takes to execute it. To get accurate metrics, you must block the execution to ensure the GPU has finished its tasks before stopping your timer.

When profiling, look for:

  • Compilation overhead: Frequent re-compilations due to shape changes.
  • Data transfer: Excessive movement between host and device.
  • GPU gaps: Periods where the GPU is idle due to host-side bottlenecks.
  • Kernel granularity: Too many tiny kernels that fail to saturate the GPU.
  • Memory pressure: Inefficient use of VRAM that forces swapping or slows down execution.