A Full-Stack Approach to Inference

OpenAI’s custom silicon, Jalapeño, is designed specifically to address the bottlenecks of modern language model inference. Unlike general-purpose hardware, Jalapeño optimizes the entire stack—chip, memory, networking, and software—to minimize data movement. By keeping model states (like KV cache) local and tightly integrating the network, the architecture maintains high performance across both compute-intensive prefill phases and memory-bandwidth-constrained decode phases. This balance is critical for agentic workloads that require rapid, sequential processing.

Performance Gains and Efficiency

Testing against leading commercial systems using the InferenceX benchmark, Jalapeño demonstrated superior performance across GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T models. Key performance metrics include:

• Efficiency: 1.5 to 1.9 times more AI work per watt at peak throughput.
• Latency: 1.7 to 3.6 times lower end-to-end latency.
• Interactive Workloads: 2.1 to 4.1 times higher performance in highly interactive scenarios.

OpenAI emphasizes that performance should be measured by useful AI work per unit of power rather than raw chip-level metrics. Jalapeño is rated at 700W but sustained workloads at or below 550W, placing it on the Pareto frontier for throughput and power efficiency.

AI-Driven Development and Programmability

OpenAI utilized its own models to accelerate the chip's development, moving from design to tapeout in just nine months. The chip was architected to be a predictable programming target, allowing AI to optimize the mapping, placement, and scheduling of workloads. This synergy extends to software development: AI-generated implementations for specific attention and mixture-of-experts blocks outperformed human-expert-written code by 1.5 to 1.8 times. This indicates a shift toward a development loop where AI actively participates in hardware-level optimization.