OpenAI's Jalapeno Chip Explained: How Custom Silicon Cuts LLM Inference Latency Up to 3.6x

Every AI product with a "generate" button is bottlenecked by the same thing behind the scenes: how fast a chip can move data in and out of memory while a model produces one token at a time. On August 25, 2026, OpenAI published the first independently measured performance results for Jalapeno, its first custom inference chip, and the numbers are specific enough to actually learn something from, not just a press release restating "faster and cheaper" [1].

This post walks through what Jalapeno actually is, why it was built the way it was, what the benchmark numbers mean in practice, and how it stacks up against Google's TPU and Amazon's Trainium, the two other major custom inference chips already in production. Along the way there is real code for the memory-bandwidth math that explains why any of this matters.

If you would rather see the mechanism than read about it, here is a short video generation prompt built around Jalapeno's core design idea, keeping data physically close to the compute that needs it, written for a Wan-style video model:

A small polished metal chili-pepper-shaped object sits on a circuit board tray as a mechanical arm lowers a glowing measuring probe toward it, six tiny drawers slide shut around it in perfect sync until they are flush against its sides, then a brass stopwatch beside the tray visibly slows its ticking hand. Clean scientific motion-graphics style, precise geometric shapes, soft pastel lighting, no readable text, no logos, no people, smooth steady camera push-in.

Step 1: What Jalapeno Actually Is

Jalapeno is OpenAI's first chip built specifically for inference, the process of running a trained model to generate a response, as opposed to training, the much more compute-heavy process of teaching a model in the first place. It is being built in partnership with Broadcom, under a roughly 10 gigawatt deployment deal the two companies signed in October 2025, with the first working silicon publicly unveiled on June 24, 2026 [2].

OpenAI's own framing for why it needed custom silicon at all is straightforward. Richard Ho, OpenAI's head of hardware, put it this way when the August benchmark results came out: "Jalapeno can serve more AI work per unit of power, while also returning responses more quickly" [3]. That single sentence contains the two axes every inference chip gets judged on: throughput, how much total work a chip can push through per unit of power, and latency, how quickly any single request comes back. Those two goals are usually in tension, since batching more requests together to raise throughput tends to slow down each individual response. Ho described Jalapeno's positioning on that tradeoff directly: "It's very efficient to serve a lot of customers, but it can also be very low latency" [3].

OpenAI is not walking away from NVIDIA. The company has said it will continue deploying NVIDIA and other partners' accelerators for both training and inference, with Jalapeno positioned as an additional, purpose-built option rather than a wholesale replacement [1]. That matters for reading this whole story correctly: this is a company that spends enormous sums on NVIDIA hardware choosing to also build something narrower and more specialized for one specific job, generating tokens as fast and cheaply as possible.

Step 2: The Numbers That Matter, and Where They Come From

The August 25 results were not self-reported in a vacuum. They were run on InferenceX, an open benchmarking framework maintained by SemiAnalysis that automates recurring, reproducible LLM serving tests, tracking token throughput, cost per token, and tokens generated per megawatt across vendors and workloads on an ongoing basis rather than as a one-time snapshot [4]. Using a third-party benchmark matters here, since a chip vendor grading its own homework is a weaker claim than a chip vendor's results being reproducible on a framework other labs and hyperscalers already use to compare NVIDIA, AMD, Google, and AWS silicon against each other.

Against that benchmark, Jalapeno delivered 1.5 to 1.9 times higher throughput per kilowatt and 1.7 to 3.6 times lower end-to-end latency than NVIDIA's GB200 and GB300 rack-scale systems, OpenAI's current highest-volume inference hardware [3] [5]. Testing reportedly covered a range of model sizes, including OpenAI's own smaller models as well as third-party open weight models from DeepSeek and Moonshot AI, which matters because a chip that only performs well on one narrow model architecture is a much less useful chip than one that generalizes [5].

How InferenceX Actually Runs a Test

It helps to know what the benchmark is doing mechanically, since "1.5x to 1.9x" means something different depending on how the test was constructed. InferenceX spins up a real inference server, running frameworks like vLLM, SGLang, or TensorRT-LLM depending on the model being tested, and points a benchmark client at it that fires requests, times the responses, and logs throughput and latency for each one [4]. Crucially, the framework runs each model across a range of input-sequence-length and output-sequence-length combinations, short prompts with long answers, long prompts with short answers, and everything in between, using randomized sequences specifically so prefix caching cannot artificially inflate the numbers for one vendor's chip over another's [4].

That range-based approach is why the reported gains are a band, 1.5x to 1.9x, rather than one clean number. A workload dominated by short decode runs behaves differently on a memory-bandwidth-bound chip than a workload dominated by long prefill passes, and a serious benchmark reports the spread rather than cherry-picking the friendliest scenario. If you ever see an inference chip claim reported as a single multiplier with no range, that is usually a sign the underlying test covered a narrower set of conditions than InferenceX-style methodology does.

Here is a simplified version of the throughput-per-kilowatt calculation that a benchmark like InferenceX is actually computing under the hood:

python
/code
def throughput_per_kw(tokens_per_second, package_watts):
    """Normalizes raw throughput by power draw, the metric SemiAnalysis'
    InferenceX benchmark and OpenAI's own Jalapeno results both report
    in, since a chip that is merely fast but power-hungry does not
    actually lower the cost of serving a model at scale."""
    kilowatts = package_watts / 1000
    return tokens_per_second / kilowatts


def relative_efficiency(jalapeno_tokens_per_kw, baseline_tokens_per_kw):
    """Reproduces the 1.5x-1.9x throughput-per-kilowatt range OpenAI
    reported for Jalapeno against NVIDIA's GB200 and GB300 rack systems
    on the InferenceX benchmark."""
    return jalapeno_tokens_per_kw / baseline_tokens_per_kw


# Example: a package rated for 700W but drawing a sustained 550W during
# testing, generating 550 tokens/sec against a GB200 baseline generating
# roughly 340 tokens/sec at comparable power.
jalapeno = throughput_per_kw(550, 550)
gb200_baseline = throughput_per_kw(340, 550)
print(round(relative_efficiency(jalapeno, gb200_baseline), 2))  # lands in OpenAI's reported 1.5x-1.9x band

Power draw is the detail that makes the 1.5 to 1.9 times figure meaningful rather than a marketing rounding. The Jalapeno package is rated for 700 watts, but sustained draw during testing stayed at or below 550 watts, meaning the reported throughput gains were measured against a chip that was not even running at its own power ceiling [5]. Richard Ho summarized the overall result plainly: "The bottom line is that the results show a very, very significant performance advance over state of the art" [3].

Step 3: Why Inference Is a Memory Problem, Not a Compute Problem

To understand why Jalapeno's specs look the way they do, it helps to understand what actually happens when a language model generates a response, because it is not what most people assume.

Generating text with a transformer happens in two distinct phases. Prefill processes the entire input prompt at once, which is compute-heavy but happens only once per request. Decode then generates one output token at a time, and each of those steps has to read the model's full weights, plus a growing cache of every previous token's key and value vectors, called the KV cache, out of high-bandwidth memory (HBM) before it can produce the next token. Decode is where most of a chat response's latency actually comes from, and it is fundamentally a memory-bandwidth problem, not a raw-compute problem, because the GPU or chip's arithmetic units sit mostly idle waiting for data to arrive from memory.

Here is roughly what that memory footprint looks like in code, and how it turns into decode latency:

python
/code
def kv_cache_bytes(num_layers, num_kv_heads, head_dim, seq_len, batch_size, bytes_per_value=2):
    """Rough KV cache memory footprint for a transformer decode step.
    Every token generated has to keep its key and value vectors resident
    in memory for every layer, so this grows linearly with sequence
    length and batch size, not with model compute. This is why decode
    is a memory-bandwidth problem, not a raw-FLOPs problem."""
    bytes_per_token = num_layers * num_kv_heads * head_dim * 2 * bytes_per_value
    return bytes_per_token * seq_len * batch_size


def decode_step_seconds(cache_bytes, model_weight_bytes, hbm_bandwidth_bytes_per_sec):
    """During autoregressive decode, generating one token requires reading
    the full model weights plus the KV cache from HBM at least once.
    A chip's decode speed is bounded by how fast it can move that many
    bytes, not by how many FLOPs it can do, which is why memory
    bandwidth (TB/s), not compute (TFLOPs), is the number that matters
    most for chips like Jalapeno, Ironwood, and Trainium3."""
    total_bytes = cache_bytes + model_weight_bytes
    return total_bytes / hbm_bandwidth_bytes_per_sec

This is the reason Jalapeno's design documentation emphasizes minimizing data movement above almost everything else: localized KV cache placement, reduced communication delay specifically during the handoff between prefill and decode, and tight coordination between compute, memory, and networking components built around those two distinct phases rather than a single generic compute pipeline [3]. A chip that is excellent at raw FLOPs but has to wait on a slower memory bus will still bottleneck on decode, which is exactly the trap a general-purpose training-first GPU architecture can fall into when it gets repurposed for inference.

A Worked Example: What the KV Cache Actually Costs

The abstract version of this argument is easy to nod along to and hard to actually feel, so it helps to run real numbers through the function from the code block above. Take a mid-sized 70 billion parameter dense model with 80 transformer layers, 64 key-value heads, and a head dimension of 128, serving a single request with a 8,000 token context, using standard 16-bit precision for the cache:

The KV cache for that one request alone works out to roughly 80 layers times 64 heads times 128 head dimension times 2 (one vector for keys, one for values) times 2 bytes times 8,000 tokens, which lands at just under 20 gigabytes for a single sequence. Multiply that by a realistic serving batch of even 32 concurrent requests and the KV cache alone reaches roughly 620 gigabytes that has to be read from HBM on every single decode step, on top of the model weights themselves. That is the concrete number behind the abstract claim that decode is memory-bound: at Trainium3's 4.9 TB/s of bandwidth, reading 620 GB takes about 127 milliseconds per step before a single new token comes out, while at Jalapeno's reported 15.4 TB/s the same read takes roughly 40 milliseconds, a difference a user would genuinely feel as the gap between a response that streams smoothly and one that visibly stutters.

Step 4: Inside the Silicon

Jalapeno's physical specs back up that memory-first design philosophy. Each package integrates six HBM4 memory stacks totaling 216 GiB of capacity, with 15.4 TB/s of memory bandwidth, manufactured on TSMC's 3nm process node [5]. Reporting indicates Samsung Electronics supplies the HBM4 stacks while Broadcom serves as the ASIC design and manufacturing partner, with OpenAI itself handling the chip architecture, meaning the compute and memory layout decisions that determine how well the chip actually performs on real inference workloads [5].

That 15.4 TB/s bandwidth figure is worth sitting with for a second, because it directly plugs into the decode latency math from Step 3. A chip with roughly twice the memory bandwidth of an older-generation part does not just serve requests twice as fast in the abstract, it specifically halves the time spent in the exact phase, decode, that determines how quickly a user actually sees a response stream back. This is also why OpenAI's own hardware team has been explicit that Jalapeno was developed with OpenAI's own models assisting in the design process itself, an unusually direct feedback loop between the software doing the generating and the silicon generating it [2].

Why HBM4 Specifically, and Why the Foundry Matters

The choice of HBM4 over the previous-generation HBM3E used in chips like Ironwood is not incidental. Each successive HBM generation roughly increases per-stack bandwidth while also increasing pin count and stacking density, which is why Jalapeno's six HBM4 stacks can hit 15.4 TB/s in a package while Ironwood's eight HBM3E stacks top out at 7.4 TB/s despite having more stacks. Being an early, large-scale adopter of HBM4 is itself a supply chain bet, since HBM4 capacity is still ramping industry-wide and Jalapeno is reportedly competing with NVIDIA's own HBM4 demand for the same Samsung production lines [5].

The TSMC 3nm process node is the other half of the equation, and it is the same node Amazon uses for Trainium3, which puts Jalapeno and Trainium3 on comparable transistor density and power efficiency footing at the silicon level, even though their memory subsystems differ substantially. That shared foundry detail is a useful reminder that a lot of what separates these chips is not exotic manufacturing, it is architectural decisions, how much HBM to attach, how tightly to couple it to compute, how to lay out the interconnect, made by teams optimizing for slightly different workloads and constraints.

Step 5: How Jalapeno Compares to Google TPU Ironwood and AWS Trainium3

Jalapeno is not the first custom inference chip from a major AI lab, it is the third. Google and Amazon both shipped their own custom silicon well before this benchmark, and putting the specs side by side is the most useful way to understand where Jalapeno actually sits in the landscape rather than taking OpenAI's framing at face value.

Google's seventh-generation TPU, Ironwood, was explicitly positioned as "the first Google TPU for the age of inference" when it launched, built for large-scale, decode-heavy serving rather than training [6]. Each Ironwood chip carries 192 GiB of HBM3E at 7.4 TB/s of bandwidth, with a full 9,216-chip pod delivering 42.5 exaflops of aggregate compute and running roughly twice as power efficient as the prior TPU generation, Trillium [7].

AWS's answer is Trainium3, its fourth-generation custom AI chip, generally available since December 2025 through EC2 Trn3 UltraServers. Each Trainium3 chip is also built on a 3nm process and delivers 2.52 petaflops of FP8 compute alongside 144 GiB of HBM3e at 4.9 TB/s of bandwidth, with a full UltraServer scaling up to 144 chips for 362 combined petaflops [8].

Lined up against those two, Jalapeno's 216 GiB and 15.4 TB/s per package is a meaningfully larger and faster memory subsystem than either Ironwood's 192 GiB and 7.4 TB/s or Trainium3's 144 GiB and 4.9 TB/s, at least on a per-chip basis. That tracks with the decode-latency math above: more bandwidth per chip means less time stalled waiting on the KV cache and model weights during exactly the phase that determines how snappy a response feels. Where Ironwood and Trainium3 have a real head start is production maturity and scale, both have been generally available and running real hyperscale workloads for months to a year, while Jalapeno is still moving from benchmark validation toward "very small volumes" by the end of 2026 and more meaningful deployment in 2027 [3].

It is also worth being honest about a limitation Richard Ho himself acknowledged: "by the time Jalapeno reaches full deployment, the competition may have advanced significantly" [3]. Chip development cycles run years, and a benchmark win today against GB200 and GB300 does not guarantee a win against whatever NVIDIA, Google, or Amazon ship next. Second-generation Jalapeno silicon is already reported to be nearing tape-out, with third-generation development underway in parallel, which is a normal cadence for a company trying to stay competitive on a multi-year hardware roadmap rather than treating this as a one-shot launch [5].

Step 6: What a Throughput-per-Kilowatt Gain Is Actually Worth

Throughput-per-kilowatt is not an abstract efficiency score, it converts almost directly into serving cost, since the electricity a data center burns to generate a million tokens is a real, recurring line item at OpenAI's scale, on top of the hardware itself. Isolating just the power side of that cost, holding electricity price and hardware amortization constant, makes the size of a 1.5x to 1.9x efficiency gain concrete rather than abstract:

python
/code
def cost_per_million_tokens(package_watts, tokens_per_second, dollars_per_kwh=0.12, hardware_amortized_dollars_per_hour=0.0):
    """Rough cost-per-million-output-tokens purely from the power side of
    the equation, holding electricity price and any amortized hardware
    cost constant, to isolate how much a throughput-per-kilowatt gain
    alone is worth. Real production cost models add hardware
    depreciation, networking, and cooling on top of this."""
    kwh_per_second = package_watts / 1000 / 3600
    seconds_per_million_tokens = 1_000_000 / tokens_per_second
    energy_cost = kwh_per_second * seconds_per_million_tokens * dollars_per_kwh
    hardware_cost = hardware_amortized_dollars_per_hour * (seconds_per_million_tokens / 3600)
    return round(energy_cost + hardware_cost, 4)


# Same 550W sustained draw, comparing 550 tok/s (Jalapeno-class) against
# 340 tok/s (GB200-class baseline) at identical power and electricity price.
jalapeno_cost = cost_per_million_tokens(550, 550)
baseline_cost = cost_per_million_tokens(550, 340)
print(jalapeno_cost, baseline_cost, round(baseline_cost / jalapeno_cost, 2))

Running that function with the same 550-watt sustained draw figure from the InferenceX results, comparing 550 tokens per second against a 340 tokens-per-second GB200-class baseline, produces roughly a 1.6x reduction in the power cost of generating a million tokens, landing squarely inside OpenAI's reported efficiency band. At OpenAI's reported inference volume, even a rough 1.5x to 1.9x reduction in the power component of serving cost compounds into a meaningful sum over a full year of traffic, which is the actual business case for spending years and a 10 gigawatt hardware commitment building custom silicon instead of simply buying more of what NVIDIA already sells [2].

This is also the same economic logic behind why Google and Amazon built Ironwood and Trainium3 in the first place, rather than running exclusively on merchant silicon. Once inference volume reaches a certain scale, a percentage-point improvement in tokens generated per watt stops being a rounding error and starts being a board-level capital allocation decision, which is exactly the scale all three of these companies are operating at.

Step 7: Why This Matters Beyond OpenAI's Own Infrastructure

It is tempting to file inference chip announcements under pure infrastructure trivia, but the effect shows up directly in what AI products can offer. Every generation step in a pipeline like Text2Shorts in Miraflow AI, which turns a topic into a script, scene visuals, and a finished vertical video, depends on multiple models responding quickly enough in sequence that the whole workflow still feels fast to the person using it. The same is true of iterating on results in the AI image generator in Miraflow AI, generating variations with the cinematic AI video generator, or scanning a long upload for viral moments with AI Clipping. None of that is about which specific chip runs underneath, it is about the fact that inference-level engineering, exactly the kind Jalapeno, Ironwood, and Trainium3 are all racing on, is what determines whether a "generate" button feels instant or sluggish. We ran into the same underlying tradeoff comparing generation speed directly in MiniMax H3 vs Veo 3.1 and again in Nano Banana Pro vs Seedream 5.0 Pro, where turnaround time mattered as much as output quality. If you want the software-side counterpart to this hardware story, our explainer on speculative decoding covers how serving engines squeeze more speed out of the exact same chips through algorithmic tricks rather than new silicon. You can browse more breakdowns like these on the Miraflow AI blog, and every tool named above lives at miraflow.ai.

Common Mistakes and Misunderstandings

A few misreadings show up often when a story like this gets summarized secondhand.

• Assuming Jalapeno replaces NVIDIA at OpenAI. OpenAI has been explicit that it will keep deploying NVIDIA and other partners' accelerators alongside Jalapeno, not instead of it [1].
• Treating the 1.5x-3.6x figures as a single number. Throughput-per-kilowatt and latency are two different metrics with two different ranges, and conflating them overstates or understates the real result depending on which one gets dropped.
• Assuming this is a training chip. Jalapeno is purpose-built for inference specifically. Training workloads remain NVIDIA's stronghold, and none of the reported benchmark results claim otherwise.
• Assuming "announced" means "deployed at scale." As of this benchmark, Jalapeno is still moving toward small-volume deployment by the end of 2026, with meaningful scale expected in 2027, not already running OpenAI's production traffic [3].
• Assuming more compute (TFLOPs) is what makes an inference chip fast. As the KV cache math in Step 3 shows, decode speed is bounded by memory bandwidth far more than by raw arithmetic throughput.

A Practical Framework for Reading Inference Chip Announcements

Whenever a new inference chip announcement like this one comes out, whether it is from OpenAI, Google, Amazon, or a smaller player, the same checklist separates a real result from a marketing headline.

• Check whether the benchmark is third-party and reproducible, like SemiAnalysis' InferenceX, versus a number the vendor generated and reported entirely on its own.
• Separate throughput-per-power claims from latency claims. They measure different things and a chip can be strong on one and weak on the other.
• Look for the power draw the benchmark actually ran at, not just the package's rated maximum, since testing below the power ceiling changes what the efficiency numbers mean.
• Check deployment stage language carefully. "Benchmarked," "small volume," and "generally available at scale" are three very different claims that get blurred together in headlines.
• Compare memory bandwidth, not just compute (FLOPs), for any workload that is decode-heavy rather than prefill-heavy, since that is almost always the real bottleneck in production LLM serving.
• Ask what the comparison baseline actually is. "Faster than NVIDIA" means something different depending on whether the baseline is a current-generation GB200/GB300 system or an older one, and Jalapeno's own numbers were specifically measured against OpenAI's current highest-volume hardware, not a strawman.
• Weigh production maturity separately from raw specs. Ironwood and Trainium3 have real production track records at scale today. A chip that wins on paper specs but has not yet run a full year of live traffic carries execution risk the spec sheet does not capture.
• Watch for the next generation. Second and third-generation Jalapeno silicon are already in development, which is normal for any serious hardware program, but it also means today's comparison is a snapshot, not a final verdict.

If you are evaluating whether any of this changes something for a product you build on top of, the practical test is simple: does your workload spend most of its time in decode, generating long streaming responses, versus prefill, processing large inputs. Chat interfaces, coding agents, and most content-generation pipelines lean heavily decode-bound, which is exactly the profile all three of these chips, Jalapeno, Ironwood, and Trainium3, are being optimized around.

Frequently Asked Questions

Is Jalapeno available to the public or to developers right now? No. As of the August 2026 benchmark results, Jalapeno is in the validation and testing stage, with OpenAI targeting very small production volumes by the end of 2026 and more meaningful deployment in 2027 [3].

Does Jalapeno replace NVIDIA GPUs for OpenAI? No. OpenAI has said it will continue deploying NVIDIA and other partners' accelerators for both training and inference, using Jalapeno as an additional purpose-built option rather than a full replacement [1].

What is the actual performance advantage over NVIDIA hardware? On the SemiAnalysis InferenceX benchmark, Jalapeno showed 1.5 to 1.9 times higher throughput per kilowatt and 1.7 to 3.6 times lower end-to-end latency compared to NVIDIA's GB200 and GB300 rack-scale systems [3] [5].

How does Jalapeno compare to Google's TPU or Amazon's Trainium chips? On paper, Jalapeno's per-chip memory subsystem, 216 GiB at 15.4 TB/s, is larger and faster than Ironwood's 192 GiB at 7.4 TB/s or Trainium3's 144 GiB at 4.9 TB/s. Google and Amazon's chips have a real head start in production maturity, having already been deployed at scale for months [6] [8].

Why does inference speed matter for AI content tools, not just chatbots? Any tool with a generate step, video, image, thumbnail, or music, depends on the same underlying memory-bandwidth-bound decode process this post describes. Faster inference chips mean faster iteration for anyone using those tools, not just faster chat responses.

Who manufactures Jalapeno? Broadcom serves as the ASIC design and manufacturing partner, Samsung Electronics reportedly supplies the HBM4 memory, and the chip is fabricated on TSMC's 3nm process, with OpenAI handling the core chip architecture [5].

Conclusion

Jalapeno is a genuinely specific result, not a vague "OpenAI builds its own chip" headline: a 1.5 to 1.9 times throughput-per-kilowatt gain and a 1.7 to 3.6 times latency reduction against NVIDIA's current GB200 and GB300 systems, measured on a third-party benchmark, backed by a memory subsystem that outpaces both Google's Ironwood and Amazon's Trainium3 on paper. What makes it worth understanding rather than skimming past is the mechanism underneath the numbers: decode-phase inference is a memory-bandwidth problem, and every one of these chips, Jalapeno included, is being architected around minimizing data movement rather than maximizing raw compute. That is the same underlying constraint that determines how fast any AI generation tool feels to actually use, whether it is a chatbot, an image generator, or a video pipeline. Jalapeno will not reach meaningful production scale until 2027, so the real test is still ahead, but the benchmark validation released this week is the first concrete evidence that OpenAI's custom silicon bet is producing real, measurable results rather than a roadmap slide. If you want to go deeper on the software-side techniques that squeeze more speed out of chips like these, our breakdown of speculative decoding is the natural next read.

References and Sources

[1] OpenAI. "OpenAI and Broadcom unveil LLM-optimized inference chip."

[2] CNBC. "OpenAI and Broadcom reveal Jalapeno, first AI chip in partnership."

[3] TechCrunch. "OpenAI's Jalapeno chip is built for fast inference at scale, benchmarks show."

[4] SemiAnalysis. "InferenceX: Open-Source Agentic Inference Benchmark."

[5] TrendForce. "OpenAI Debuts Jalapeno AI Inference Chip, with Samsung Reportedly Supplying HBM4."

[6] Google. "Ironwood: The first Google TPU for the age of inference."

[7] Google Cloud Blog. "Inside the Ironwood TPU codesigned AI stack."

[8] AWS. "Announcing Amazon EC2 Trn3 UltraServers for faster, lower-cost generative AI training."