Netflix’s LLM Serving Platform: A Deep Dive into Triton and vLLM

Think Netflix is just about streaming? Behind the scenes, they run a massive ML infrastructure powering recommendations, search, and even content production. Recently, their engineers shared how they built an internal LLM serving platform using Triton and vLLM. The post is a goldmine for anyone wrestling with production LLM inference—especially if you're balancing performance, flexibility, and operational sanity.

The Problem: Not All Models Fit the Same Mold

Netflix doesn't run just one LLM. They have models of various sizes, each with different hardware needs and latency requirements. Some are small enough for CPUs, while larger ones demand GPUs. And the inference engine landscape is shifting fast. You start with one framework, a better one comes along, and you want to switch without rewriting your entire serving layer.

The challenge: support this diversity without a mess of bespoke solutions. Netflix wanted a unified platform handling real-time and batch workloads on both CPU and GPU, while letting underlying engines evolve independently.

The Architecture: JVM at the Core, Triton and vLLM at the Edges

Netflix's existing JVM-based service layer handles the heavy lifting around model inference: routing, feature fetching, candidate generation, post-processing, logging. This is where the magic of integration happens. For small models, they run inference directly in the JVM process on CPU. For larger ones, they delegate to a model serving service (MSS) that uses Triton for model loading, batching, GPU scheduling, and multi-framework serving.

This design keeps peripheral production workflows consistent, even when inference happens on remote hardware. The JVM layer doesn’t care whether the model runs locally or on a GPU box—it just sends requests and gets responses. That abstraction is powerful.

Why vLLM for GPU Execution?

For the GPU path, Netflix chose vLLM for its operational adaptability and scalability. But they didn't ditch Triton entirely. Instead, Triton manages the model and scheduling, while vLLM handles actual inference and provides extension points for custom behavior.

It's a clean division of labor: Triton wraps the model and manages the serving environment; vLLM does the heavy lifting and lets you plug in custom code. This separation is key because it means you can swap out the inference engine without touching the rest of the serving stack.

But this isn't without friction. Netflix found that mismatched versions of Triton and vLLM can cause deployments to fail outright. So they now pin tested versions together and test them as a pair. Small operational detail, but it can save you hours of debugging.

Custom Models and Hugging Face Compatibility

One of the biggest headaches in production LLM serving is dealing with custom models. Netflix has models that don’t fit neatly into Hugging Face's compatibility layer, which vLLM relies on. To handle these, they use vLLM's extension points to support custom architectures and decoding behaviors.

This is where the rubber meets the road. If you're running open-source models off the shelf, you're fine. But if you've trained something with a twist, you'll need to dig into the engine's internals. Netflix’s experience shows you can't always rely on standard tooling—you need to be willing to get your hands dirty.

Choosing the Right Triton Backend: Python vs. vLLM

Netflix compared two ways to package models with Triton: the Triton Python backend and the vLLM backend. They found the vLLM-backend approach lets the model and front-end evolve more independently than the Python-backend method. This choice affects how tightly coupled the model is to its serving environment, not which engine executes inference.

In other words, you can have vLLM as the engine but still use the Python backend for packaging—but then you might tie yourself to a specific serving setup. The vLLM backend gives more flexibility in the long run, even if it seems more complex initially.

The Illusion of a Unified Interface

Netflix exposes a common serving interface, including an OpenAI-compatible API and KServe's HTTP and gRPC frontends. But the team quickly learned a unified interface doesn't mean underlying engines behave identically. They ran into differences in how features are handled across these integrations.

For example, constrained decoding—forcing the model to generate responses in a specific format, like valid JSON—requires the decoder to maintain state across the entire request. When vLLM pauses a request to manage GPU resources and then resumes it, that state can get out of sync with token history. Netflix had to add logic to detect these changes and rebuild state before continuing generation.

Subtle but critical. Even with a clean abstraction, you still need to understand the quirks of each engine.

Deployment Strategies: Red-Black and Versioned

Deploying model updates without breaking consumers is a classic problem. Netflix uses Red-Black and Versioned deployment strategies to handle changes at the model level. Red-Black is a simple canary approach, but Versioned deployments are more interesting: they keep old and new versions of a model running in parallel, allowing consumers to migrate gradually as they adapt to incompatible input or output schemas.

This is especially useful when a model update changes the API contract. Instead of forcing all consumers to update at once, they can take their time. Pragmatic, acknowledging the reality of a complex microservices ecosystem.

Comparing with Uber's Approach

Uber has described a similar pattern at the application boundary. Their generative AI gateway provides an OpenAI-compatible interface between externally hosted and internally managed models, centralizing concerns like authentication, caching, observability, and routing. The implementation differs from Netflix's service platform, but both separate application integration from the backend model, runtime, and hosting environment.

Common theme: you want a stable interface for your application teams, but you also want the freedom to swap out the model or runtime without breaking everything. Both companies arrived at the same conclusion, albeit through different paths.

Lessons Learned: The Abstraction Doesn't Remove the Work

Netflix's experience is a reminder that a general-purpose serving interface can sit on top of several different layers. The architecture aims to provide a stable integration surface for application teams while allowing model providers and service runtimes to evolve. But the abstraction doesn't eliminate the underlying work: packaging, compatibility control, constrained decoding, and deployment isolation still need to be engineered at every layer.

If you're thinking about building your own LLM serving platform, take note: it's not just picking a framework and calling it a day. You need to think about version pinning, custom model support, decoding constraints, and deployment strategies. Even then, you'll likely encounter edge cases requiring custom logic.

Netflix’s post is a candid look at the messy realities of production machine learning. It’s not a pretty picture, but it’s honest. And for anyone in the trenches, that’s more valuable than a polished tutorial.