# Modelplane Documentation

> Modelplane is the open source control plane for AI model serving. It extends Crossplane to manage AI inference across a fleet of GPU clusters.

---

# All recipes

Source: /recipes/all/

Every validated recipe in one table: model, size, architecture, precision, and
the verified hardware. Select a row for the full recipe.






  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  

  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  

  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  

  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  

  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  

  
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  
    
  



  

  

  

  

  

  

  

  

  


  
    <thead>
      <tr>
        <th>Model
        <th>Size
        <th>Arch
        <th>Precision
        <th>Verified on
        <th>Notes
      
    
    <tbody>
      
        
        
        
        <tr>
          
            
              



              
                Qwen3-8B
                qwen
              
            
          
          8B
          <td>Dense
          BF16
          
            EKS
            <strong>L4
          
          An 8.2B dense chat model on a single NVIDIA L4.
        
      
        
        
        
        <tr>
          
            
              



              
                Qwen2.5-7B
                qwen
              
            
          
          7B
          <td>Dense
          AWQ INT4
          
            Vultr
            <strong>A16
          
          A 7B dense chat model (AWQ INT4) on a single NVIDIA A16 on Vultr.
        
      
        
        
        
        <tr>
          
            
              



              
                Qwen3-Coder-480B
                qwen
              
            
          
          480B A35B
          <td>MoE
          BF16 / FP8
          
            EKS
            <strong>H200
          
          A 480B code MoE, multi-node BF16 over EFA or single-node FP8 on SGLang.
        
      
        
        
        
        <tr>
          
            
              



              
                Qwen2.5-72B
                qwen
              
            
          
          72B
          <td>Dense
          AWQ INT4
          
            AKSNebius
            <strong>A100<strong>H100
          
          A 72B dense chat model (AWQ INT4) on a single 80 GB GPU, on AKS and Nebius.
        
      
        
        
        
        <tr>
          
            
              



              
                Kimi-K2
                moonshotai
              
            
          
          1T A32B
          <td>MoE
          INT4
          
            EKS
            <strong>H200
          
          A 1T MoE served prefill/decode disaggregated across two H200 nodes.
        
      
        
        
        
        <tr>
          
            
              



              
                Llama-3.1-8B
                meta-llama
              
            
          
          8B
          <td>Dense
          BF16
          
            EKSGKE
            <strong>L4
          
          An 8B dense chat model on a single NVIDIA L4.
        
      
        
        
        
        <tr>
          
            
              



              
                GLM-4.5-Air
                zai-org
              
            
          
          106B A12B
          <td>MoE
          GGUF IQ4_XS
          
            GKE
            <strong>A100
          
          A 106B MoE served from a GGUF checkpoint via llama.cpp on a single A100.
        
      
        
        
        
        <tr>
          
            
              



              
                Nemotron-3.5-Lightning
                nvidia
              
            
          
          30B A3B
          <td>MoE
          NVFP4
          
            Nebius
            <strong>H100
          
          An open 30B MoE with 3B active parameters served NVFP4 on a single H100 on Nebius.
        
      
        
        
        
        <tr>
          
            
              



              
                Laguna-S-2.1
                poolside
              
            
          
          118B A8B
          <td>MoE
          FP8
          
            Nebius
            <strong>H100
          
          A 118B code MoE served FP8 on a single 8x H100 node on Nebius.
        
      
    
  




---

# Overview

Source: /overview/



Modelplane is the open source control plane for AI inference. It's software you
install and run in your own environment, and it orchestrates the models, serving
stack, and infrastructure across cloud, neocloud, and on-premise. Modelplane
supports running any model and any engine on any infrastructure, with the
frontier-level serving topologies and performance the largest models demand,
from a single GPU to disaggregated, multi-node deployments.

Modelplane operates across the whole fleet: provisioning inference clusters,
scheduling model deployments on compatible clusters, autoscaling model replicas
across clusters, caching model weights across clusters, and routing across
clusters.

It's an active system that is always reconciling the fleet toward the state you
declare. You install Modelplane on a Kubernetes cluster, which becomes the
control cluster for your inference fleet. It's built on
[Crossplane](https://crossplane.io) and fully integrates with your existing
platform systems.




  
    
    Warning
  
  
    Modelplane is under active development. We have opted to build the project in the
open, collaborating with the broad AI inference community on integrations and
capabilities.
  



## Deploy a model

Modelplane's API is declarative, designed for platform teams responsible for the
inference infrastructure and developers deploying models on that infrastructure.

Once a platform team has provisioned inference clusters and declared the available
GPUs and networking fabric, an ML development team deploys a model with a
declarative manifest:


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args: ["--model=Qwen/Qwen2.5-0.5B-Instruct"]

  



Modelplane schedules a model replica onto an inference cluster with free,
compatible GPUs and memory, and deploys the serving engine. Exposing an
OpenAI-compatible endpoint can be done by declaring a model service:


  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-demo

  



## A universal control plane for AI inference


Modelplane is designed to be a universal control plane for inference. It runs
inference clusters on any cloud, neocloud, or on-premise environment, or any
combination of them. Modelplane can provision the clusters for you, or you can
bring your own.

It supports any serving engine that runs as a container, and can serve
frontier-quality models using advanced topologies including tensor parallel,
pipeline parallel, data and expert parallel, and prefill/decode disaggregation.
Modelplane works across different accelerators and networking fabrics, and
schedules each model's replicas by matching the model's hardware requirements to
the hardware available across your clusters.

## What Modelplane is not

Modelplane is not a serving engine like vLLM, SGLang, or TensorRT-LLM. Modelplane
composes serving engines and orchestrates them fleet-wide across cloud, neocloud,
and on-premise. Modelplane is not a managed inference service like Baseten,
Together, or Fireworks. These offer cloud services, while Modelplane is
self-hosted software.

## Next steps




  
    
    Get started
  
  Go from nothing to a live OpenAI-compatible endpoint in about 45 minutes.
  Deploy on a real fleet →



  
    
    Why Modelplane
  
  Learn more about Modelplane’s capabilities and how it works.
  Learn more →










---

# Anthropic Messages API

Source: /guides/anthropic-messages-api/


A vLLM server registers the Anthropic Messages API at `/v1/messages` alongside
its OpenAI routes, with no extra flag. Modelplane's route matches the
`/<namespace>/<service>/` prefix and preserves the path below it, so the same
service URL answers both `/v1/chat/completions` and `/v1/messages`. A client that
speaks the Messages API, including Claude Code via `ANTHROPIC_BASE_URL`, talks to
the deployment directly. See
[Alternate APIs](/models/model-service/) for the routing detail.

This recipe serves Qwen3-8B on a single NVIDIA H100 on Nebius, with tool calling
on: `--enable-auto-tool-choice` and `--tool-call-parser=hermes` are what let
Claude Code's tool use work. An 8B model needs a fraction of an H100, so the GPU
has ample headroom. Apply the platform side first, then the ML side.

## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for a single-H100 Nebius shape, serving Qwen3-8B.
#
# Nebius sizes nodes by platform + preset rather than an instance type; the
# preset determines the GPU, vCPU, and memory shape of each node. The devices
# block describes what a node of this class has, DRA-style - used by the
# scheduler to match models to clusters, and to form DRA ResourceClaims for
# claim: DRA devices. An 8B model needs a fraction of an H100, so this shape has
# ample headroom; a single L40S (gpu-l40s-a) is the economical alternative.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: h100-1x
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100 80GB's real usable VRAM, what the NVIDIA DRA driver reports,
      # not its nominal 80GB. A nodeSelector asking for >= 80Gi would never bind.
      memory: { value: "81559Mi" }

  




  
    
      inference-cluster.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceCluster backed by a Nebius mk8s cluster. Modelplane provisions the
# full mk8s cluster and installs the inference stack; only the GPU node pool -
# referencing the InferenceClass above - is declared here. It authenticates to
# the cluster with the credentials of the Nebius ClusterProviderConfig named
# default, the same identity that provisions it.
#
# Delete with foreground cascading deletion for a clean teardown, so the
# inference stack uninstalls before the cluster's API server goes away:
#   kubectl delete inferencecluster nebius-eu-north --cascade=foreground
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: h100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  



## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen3-8B served on a single NVIDIA H100 (Nebius) by vLLM, exposed on the
# Anthropic Messages API. vLLM's server registers /v1/messages (and
# /v1/messages/count_tokens) alongside its OpenAI routes with no extra flag, so
# this is an ordinary serve: the Anthropic surface comes for free, and
# Modelplane preserves the path below the service prefix to reach it.
#
# The tool-calling flags are what make it usable from Claude Code, not decoration:
#
#   --enable-auto-tool-choice with
#   --tool-call-parser=hermes        parse the model's tool calls so Claude Code's
#                                    tool use works (qwen3_xml is for Qwen3-Coder,
#                                    not this dense model). Qwen3's tool-use
#                                    template ships in the tokenizer, so no
#                                    --chat-template is needed.
#   --reasoning-parser=qwen3 with
#   --default-chat-template-kwargs   turns thinking off. Qwen3 thinks by default,
#                                    burying a one-line answer under a <think>
#                                    block and forbidding greedy decode.
#   --max-model-len=40960            Qwen3-8B's native context (past it needs YaRN).
#                                    Claude Code reserves 32000 output tokens, so a
#                                    smaller window 500s ("max_completion_tokens
#                                    cannot be greater than max_model_len", then
#                                    input+output overflow). The client must also
#                                    cap output with CLAUDE_CODE_MAX_OUTPUT_TOKENS
#                                    so input+output fit here. H100 has KV room.
#   --gpu-memory-utilization         headroom, not correctness.
#
# The v0.23.0 image is >0.17.1, so vLLM handles Claude Code's attribution header
# without breaking prefix caching. No --port or --host: Modelplane's routing
# expects the engine on its default :8000 with a /health probe, and passes args
# through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      # No clusterSelector: the single Nebius cluster is matched on device
      # capacity alone.
      engines:
      - name: qwen3-8b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - "--model=Qwen/Qwen3-8B"
                - "--served-model-name=qwen"
                - "--max-model-len=40960"
                - "--gpu-memory-utilization=0.92"
                - "--reasoning-parser=qwen3"
                - "--default-chat-template-kwargs={\"enable_thinking\": false}"
                - "--enable-auto-tool-choice"
                - "--tool-call-parser=hermes"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the qwen3-8b deployment as a single URL. The route matches the
# /<namespace>/<service>/ prefix and preserves the path below it, so the engine's
# Anthropic Messages API at .../v1/messages rides the same address as its
# OpenAI-compatible .../v1/chat/completions. Read the public address from
# status.address:
#   kubectl get ms qwen3-8b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b

  



## Send a request

Read the endpoint's public address from the `ModelService` status:

```bash
ADDRESS=$(kubectl get ms qwen3-8b -n ml-team -o jsonpath='{.status.address}')
```

Post to `/v1/messages` in the Messages API shape. The `model` field is the
engine's `--served-model-name` (`qwen`); `max_tokens` is required:

```bash
curl "$ADDRESS/v1/messages" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "qwen",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

`status.address` is the in-cluster gateway address, so run the request from
inside the cluster if your shell can't reach it directly:

```bash
kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -s "$ADDRESS/v1/messages" \
  -H "Content-Type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -d "{\"model\":\"qwen\",\"max_tokens\":1024,\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}]}"'
```

## Point Claude Code at it

Claude Code appends `/v1/messages` to `ANTHROPIC_BASE_URL`, so point it at the
service address and map its model tiers onto the served name. vLLM doesn't check
the auth token, so any non-empty value works. Claude Code reserves 32000 output
tokens by default, which alone leaves little context room on a small model; cap
it with `CLAUDE_CODE_MAX_OUTPUT_TOKENS` so the input and output fit under the
engine's `--max-model-len` (40960 here):

```bash
export ANTHROPIC_BASE_URL="$ADDRESS"
export ANTHROPIC_AUTH_TOKEN=dummy
export ANTHROPIC_DEFAULT_OPUS_MODEL=qwen
export ANTHROPIC_DEFAULT_SONNET_MODEL=qwen
export ANTHROPIC_DEFAULT_HAIKU_MODEL=qwen
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=8192
claude
```

The gateway must be reachable from wherever `claude` runs. If `$ADDRESS` is
in-cluster only, forward the Traefik gateway service to a local port:

```bash
kubectl -n traefik-system port-forward svc/traefik 8080:80
```

Then point `ANTHROPIC_BASE_URL` at the local port, keeping the service's
`/<namespace>/<service>` path prefix so Claude Code's `/v1/messages` still routes:

```bash
export ANTHROPIC_BASE_URL="http://localhost:8080/ml-team/qwen3-8b"
```



---

# Deploy a Model

Source: /models/model-deployment/

**API:** [`modelplane.ai/v1alpha1` · ModelDeployment](/reference/modeldeployments/)

A `ModelDeployment` is the ML team's primary interface. You describe the model
you want served, the hardware it needs, and how many copies to run; Modelplane
schedules it onto matching clusters and keeps it running. You never name a
cluster.

Modelplane is unopinionated about the engine itself. You bring the container and
its flags, and Modelplane shapes a serving topology around it. The engine flags
you write carry parallelism, quantization, and KV transfer, never injected by
Modelplane.

A deployment's replica shape lives under `spec.template` (mirroring a
Kubernetes Deployment): the replica count is `spec.replicas`, labels for the
composed ModelReplicas and ModelEndpoints go on `spec.template.metadata.labels`,
and everything else lives in `spec.template.spec`. Its
`spec.template.spec.engines` array describes the topology through two choices:

- **One pod or a gang**: whether an engine is a single `Standalone` pod or a
  `Leader` with one or more `Worker` pods coordinating across nodes.
- **Unified or disaggregated**: whether `spec.template.spec.serving.mode` keeps
  prefill and decode together (`Unified`, the default) or splits them across two
  engines (`PrefillDecode`).

How many of each to run is a separate question, covered in
[Sizing a deployment](#sizing-a-deployment).

## Single-node

The default, and what the [getting started tour](/getting-started/)
deploys. One `Standalone` member is one pod on one node, claiming that node's
GPUs through its `nodeSelector`. It's usually the right choice when a model fits
on a single node. Within a node, tensor parallelism is an engine flag
(`--tensor-parallel-size`), not a Modelplane concept.

```yaml {nocopy=true}
engines:
- name: qwen
  members:
  - role: Standalone        # one pod, one node
```

## Multi-node

When a model is too large for one node's GPUs, make the engine a gang: a `Leader`
and a `Worker` whose `worker.nodes` expands to that many worker pods, one per
node. The pods serve the model together; how the model splits across them
(tensor, pipeline, data, or expert parallelism) is up to your engine flags.

A gang should use a [`ModelCache`](/models/model-cache/) via
`spec.template.spec.modelCacheRef`, so every pod mounts the same weights instead
of each pulling its own.

```yaml {nocopy=true}
modelCacheRef:
  name: qwen3-coder         # recommended for gangs
engines:
- name: qwen3-coder
  members:
  - role: Leader
  - role: Worker
    worker:
      nodes: 1              # one worker pod per node
```

A member's `env` can read pod fields through `valueFrom.fieldRef`, like setting
vLLM's `VLLM_HOST_IP` from `status.podIP`, which multi-NIC RDMA nodes need so the
engine binds the right interface instead of guessing it.

## Disaggregated serving

The prefill and decode phases have opposite hardware profiles, and on one engine
a prefill burst stalls the decodes already running. Set
`spec.template.spec.serving.mode: PrefillDecode` to run them as two engines, one
marking `phase: Prefill` and the other `phase: Decode`. Modelplane fronts the
pair with inference-aware routing that sequences prefill then decode, moving the
KV cache between them. Each phase can sit on the GPU class that suits it.

```yaml {nocopy=true}
serving:
  mode: PrefillDecode       # the two engines below are one P/D pair
engines:
- name: prefill
  phase: Prefill
- name: decode
  phase: Decode
```

Disaggregation pays off for large models under load with strict latency targets
and long context. For small models or low traffic, the KV-transfer overhead
outweighs the benefit, so unified serving is the default.

It requires an engine image that includes the **NIXL** KV-transfer runtime.
vLLM's `NixlConnector` (and SGLang's prefill/decode transfer) import the `nixl`
package, so disaggregated engines crash at startup with `NIXL is not available`
on an image that lacks it. Recent vanilla `vllm/vllm-openai` images include NIXL,
so pin a current tag rather than an old one. The engine image is yours to choose,
so this is a prerequisite Modelplane does not bundle for you.

## Requesting GPUs

You don't name a cluster or a GPU model. Instead each member's `nodeSelector`
lists the hardware its pods need, and Modelplane finds a node pool that has it.
The platform team publishes node pools as `InferenceClass` resources, each
describing the devices its nodes carry. Your request is matched against them.

A request names a device (`gpu`), how many of it each pod needs (`count`), and
one or more `selectors` the device must match:

```yaml {nocopy=true}
nodeSelector:
  devices:
  - name: gpu
    count: 1                # one GPU per pod
    selectors:
    - cel: |
        device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0
```

Each selector is a single line of [CEL](https://cel.dev/), a small expression
language, that returns true or false for one device. The part in brackets, `"gpu.nvidia.com"`, is the
GPU vendor's driver. The fields after it, like `memory` or `architecture`, are
what the platform team published for that device. This one says "match a GPU
whose memory is at least 40Gi." A device has to match every selector in the
request. Give two selectors to mean "Hopper, with at least 80Gi."

### Requesting more than one device

`devices` is a list, so a member can ask for distinct kinds of hardware at once,
each its own entry with its own `count` and `selectors`. A node pool matches the
member only when it satisfies every entry. This is how you ask for both a GPU and
a fast NIC on the same node:

```yaml {nocopy=true}
nodeSelector:
  devices:
  - name: gpu
    count: 8
    selectors:
    - cel: device.attributes["gpu.nvidia.com"].architecture == "Hopper"
  - name: nic
    count: 1
    selectors:
    - cel: device.attributes["nic.nvidia.com"].linkType == "infiniband"
```

### What you can match on

Each selector is evaluated against one device and must return a boolean. The
device exposes three things:

- `device.driver`: the device's driver, a string.
- `device.attributes["<driver>"].<name>`: a typed attribute (string, bool, int,
  or version), such as `architecture` or `cudaComputeCapability`.
- `device.capacity["<driver>"].<name>`: a capacity quantity, such as `memory`.

Two helpers build comparable values: `quantity()` parses Kubernetes quantities
like `"40Gi"`, and `semver()` parses versions like `"9.0.0"`. Both support
`compareTo` (which orders two values), `isGreaterThan`, and `isLessThan`. Combine
selectors with the usual CEL operators (`==`, `!=`, `>=`, `&&`, `||`).

```yaml {nocopy=true}
selectors:
# Capacity: at least 40Gi of GPU memory. >= 0 reads as "left is at least right".
- cel: device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0
# Attribute equality: a specific architecture.
- cel: device.attributes["gpu.nvidia.com"].architecture == "Hopper"
# Version attribute: a minimum CUDA compute capability.
- cel: device.attributes["gpu.nvidia.com"].cudaComputeCapability.isGreaterThan(semver("8.9.0"))
# Driver: match any device from a given driver.
- cel: device.driver == "gpu.nvidia.com"
# Presence: only match a device that publishes a given domain.
- cel: '"gpu.nvidia.com" in device.attributes'
# Two conditions in one selector.
- cel: |
    device.attributes["gpu.nvidia.com"].architecture == "Hopper" &&
    device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("80Gi")) >= 0
```

This is the Kubernetes DRA device selector expression surface. The
Kubernetes-specific CEL extension libraries (such as regular expressions and IP
address helpers) aren't available. Selectors in practice are attribute and
capacity comparisons like those above.

### Seeing what's available

To see what you can match against, list the classes the platform team has
published and look at the devices each one declares:

```bash
kubectl get inferenceclass
kubectl describe inferenceclass gke-l4-1x-g2
```

The `describe` output shows each device's driver, attributes (like
`architecture`), and capacity (like `memory`), which are exactly the keys your
selectors read. If a selector asks for something no published class offers, the
deployment won't schedule.

## Sizing a deployment

Three independent numbers control how many pods a deployment runs:

- **`spec.replicas`** stamps out whole copies of the entire topology. Each
  replica is a complete serving instance, and replicas usually land on different
  clusters. This is the scaling axis (see [Scaling](#scaling)).
- **`engines[].copies`** runs several identical copies of one engine within a
  replica, on the same cluster. It's a fixed number, sized once, never
  autoscaled. Copies make a replica more resilient within its cluster: a node
  failure drops one copy instead of taking the whole replica out of service. In
  disaggregated serving they also set the prefill-to-decode ratio.
- **`worker.nodes`** sets how many nodes one gang spans: a `Leader` plus that
  many `Worker` pods. It's how big a single multi-node engine is.

## Scaling

`spec.replicas` is the only scaling axis. Each replica is a complete,
fixed-shape serving instance, so scaling adds or removes whole instances across
the fleet. Because the deployment exposes the Kubernetes scale subresource,
`kubectl scale` and KEDA work without anything extra. There's no in-cluster pod
autoscaling.

## Choosing a topology

| Topology | Use when | How you set it |
|----------|----------|----------------|
| Single-node | The model fits on one node's GPUs | One `Standalone` member (the default) |
| Multi-node | The model is too large for one node | A `Leader` and one or more `Worker` members, ideally with a `modelCacheRef` |
| Disaggregated serving | Large model, heavy load, strict latency, long context | `serving.mode: PrefillDecode` with two phase engines |

## Examples










  
  
  
  
    Single-node
    
    
  
  
  
  
  
    Multi-node
    
    
  
  


  








  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelDeployment deploys a model to one or more inference clusters.
# The scheduler picks clusters by clusterSelector labels and nodeSelector
# device requests, gated on available nodes. Each matched cluster gets one
# ModelReplica.
#
# The control plane creates a unified OpenAI-compatible endpoint:
#   http://<gateway-address>/<namespace>/<name>/v1/chat/completions
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  # Number of ModelReplicas to fan out to. Each replica is a complete
  # serving instance scheduled to one InferenceCluster.
  replicas: 1

  # Optional: restrict the scheduler to clusters with specific labels.
  # clusterSelector:
  #   matchLabels:
  #     modelplane.ai/region: us-central

  # Engines are an array of inference engines. This model is one engine, one
  # Standalone member, one pod - the simplest shape. The engine composes to a
  # Deployment fronted by a Service.
  template:
    spec:
      engines:
      - name: qwen3-8b
        members:
    # A Standalone member is a single self-contained engine pod. Its template
    # carries the container named "engine" - the inference engine; its image,
    # command, and args pass through verbatim.
        - role: Standalone
      # The member's per-node device request: a list of DRA device requests
      # describing what each of the member's pods needs from its node. The
      # scheduler matches each against a candidate pool's InferenceClass
      # devices and pins the member to a pool that satisfies them. Each
      # request's CEL is real DRA CEL over a single device; quantity() and
      # semver() are helpers. claim: DRA devices also become requests in the
      # DRA ResourceClaim the serving pods claim GPUs through, so an engine
      # must declare the GPUs it needs.
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # Qwen3-8B fits comfortably on an L4; 20Gi selects one without
          # over-constraining. A larger model would ask for more memory or a
          # specific architecture here. This CEL is real DRA CEL: the scheduler
          # matches it against the pool's declared device, and DRA matches it
          # again against the GPU's ResourceSlice when it binds the claim.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - "--model=Qwen/Qwen3-8B"
                - "--served-model-name=qwen"
                - "--reasoning-parser=qwen3"
                - "--default-chat-template-kwargs={\"enable_thinking\": false}"
                - "--enable-auto-tool-choice"
                - "--tool-call-parser=hermes"

  











  
    
      model-deployment-multinode.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelDeployment serving one model across two nodes.
#
# When a model is too large to fit on one node's GPUs, make an engine a gang:
# give it a Leader and a Worker member, whose worker.nodes expands to that many
# worker pods, one per node. The scheduler picks a cluster with a pool that has
# enough GPUs per node and enough nodes for the whole gang, and Modelplane
# composes a LeaderWorkerSet-backed serving instance on it. A member joins the
# leader through $(MODELPLANE_LEADER_ADDRESS) and takes its rank from
# $(MODELPLANE_RANK) (0 on the Leader, 1..worker.nodes on the followers), both
# injected by Modelplane - so the same Worker template fans out to any number
# of nodes.
#
# Multi-node engines require a ModelCache: every pod in the gang mounts it at
# /mnt/models. When a member brings its own command, Modelplane does not inject
# --model, so the leader points the engine at the mount explicitly.
#
# This shape (vLLM's native multiprocessing backend, TP within a node and PP
# across nodes) is the one validated serving Qwen3-Coder-480B; see
# examples/qwen3-coder/ for the full platform side.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: qwen3-coder
      engines:
      - name: qwen3-coder
        members:
        - role: Leader
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve /mnt/models
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=$(MODELPLANE_RANK)
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --max-model-len=32768
                  --port=8000
        - role: Worker
          worker:
            nodes: 1
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve /mnt/models
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=$(MODELPLANE_RANK)
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --headless
                  --max-model-len=32768

  









---

# Get started

Source: /getting-started/


Modelplane is an open source control plane for AI inference. It separates two
concerns: a platform team managing GPU capacity, and ML teams deploying models
against it. Without it, every change on one side creates work for the other.
When the platform team updates infrastructure, ML teams have to react. When
model requirements change, the platform team gets a request.

With Modelplane, the platform team publishes hardware without knowing what
models will run on it. The ML team declares what a model needs without knowing
what clusters exist. The control plane resolves it and keeps it current as
both sides change.

In this tour, you'll switch between provisioning infrastructure and declaring a
model to see how they interact. By the end you'll have a GPU fleet across three regions and one OpenAI-compatible endpoint routing to a model served across two of them.

This is not a production setup and takes around 45 minutes to run.

## What you'll build

The platform team provisions a starter cluster and grows it to two larger-GPU
regions; the ML team serves a model on the L4, then scales it onto a bigger GPU,
all behind one endpoint.


<style>
   
  #asciinema-b2b29fdb8e .ap-player {
    --term-color-background: #001D2F;
    --term-color-foreground: #D7E9EE;
  }


  


<script>
  AsciinemaPlayer.create("/getting-started/what-youll-build.cast", document.getElementById("asciinema-b2b29fdb8e"), {
    autoPlay: false,
    controls: true,
    fit: "width",
    idleTimeLimit:  2 ,
    terminalFontSize: "medium",
    poster: "npt:2:13",
    theme: "monokai"
  });


## Before you begin

You'll need [kind](https://kind.sigs.k8s.io/),
[kubectl](https://kubernetes.io/docs/tasks/tools/), and
[Helm](https://helm.sh/docs/intro/install/) installed, plus an AWS or GCP account
with permission to create clusters. Each step covers what it needs as you reach
it.

## The tour

1. [Installation](/getting-started/installation/): stand up the Modelplane control plane.
2. [Build the platform](/getting-started/build-the-platform/): provision your first GPU cluster.
3. [Deploying a model](/getting-started/deploying-a-model/): serve a model and send it a request.
4. [Scale the platform](/getting-started/scale-the-platform/): grow to a multi-region fleet.
5. [Scale the model](/getting-started/scale-the-model/): serve the model from two regions behind one endpoint.

First, follow the [Installation](/getting-started/installation/) guide.


---

# Installation

Source: /getting-started/installation/

The control plane is where everything in Modelplane runs. In this step you'll install it on a local kind cluster, using Crossplane for reconciliation and the Modelplane APIs. No cloud yet, that comes next.

This step takes about five minutes.

## Prerequisites

Install [kind](https://kind.sigs.k8s.io/),
[kubectl](https://kubernetes.io/docs/tasks/tools/), and
[Helm](https://helm.sh/docs/intro/install/) on your machine.




  
    
    Important
  
  
    <p>You can run your Modelplane control plane anywhere. This tour uses kind for
illustration.
<p><strong>Make sure your container engine is configured to allow higher resource allocations.
<p>Update your Docker memory limit to <strong>8 GB. For more information, review
the Docker documentation.

  



## Install the control plane


Crossplane provides the reconciliation engine and package management. Create the
kind cluster and install it with Helm:

```bash
# Pin to kind v0.30.0 default image (containerd 2.1.4)
# kind v0.31+ ships containerd 2.2.0 which breaks Modelplane
kind create cluster --name modelplane \
  --image kindest/node:v1.34.0@sha256:7416a61b42b1662ca6ca89f02028ac133a309a2a30ba309614e8ec94d976dc5a
```

```bash
helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update crossplane-stable
helm install crossplane crossplane-stable/crossplane \
  --namespace crossplane-system --create-namespace \
  --set "args={--enable-dependency-version-upgrades}" \
  --set-json 'provider.defaultActivations=[]' \
  --wait
```

Apply the bootstrap resources. They grant Crossplane the permissions it needs to
manage your cluster:

```shell
kubectl apply -f /examples/getting-started/prerequisites.yaml
```






  
    
      
        Review the prerequisites manifest
      
    
    
      
        
  
    
      prerequisites.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Modelplane prerequisites. Apply once after installing Crossplane.
#
# These resources grant Crossplane and provider-helm the permissions
# needed to compose Gateway API, MetalLB, and Service/EndpointSlice
# routing resources. They cannot be self-composed because Crossplane
# needs the permissions before it can compose anything.
---
apiVersion: v1
kind: Namespace
metadata:
  name: modelplane-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: crossplane-compose-modelplane
  labels:
    rbac.crossplane.io/aggregate-to-crossplane: "true"
rules:
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["*"]
# Selectorless Service plus EndpointSlice composed by ModelEndpoint to route
# the control plane gateway to a remote model endpoint.
- apiGroups: [""]
  resources: ["services"]
  verbs: ["*"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["*"]
- apiGroups: ["gateway.networking.k8s.io"]
  resources: ["gateways", "gatewayclasses", "httproutes"]
  verbs: ["*"]
- apiGroups: ["gateway.envoyproxy.io"]
  resources: ["backends"]
  verbs: ["*"]
- apiGroups: ["metallb.io"]
  resources: ["ipaddresspools", "l2advertisements"]
  verbs: ["*"]
- apiGroups: ["protection.crossplane.io"]
  resources: ["usages"]
  verbs: ["*"]
---
# Give provider-helm a deterministic SA name so we can grant it
# permissions. Without this, the SA name has a random hash.
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: provider-helm-modelplane
spec:
  serviceAccountTemplate:
    metadata:
      name: provider-helm-modelplane
---
# Grant provider-helm cluster-admin. It installs full Helm charts
# (MetalLB, Envoy Gateway, LeaderWorkerSet, etc.) that create arbitrary
# resource types across namespaces.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: provider-helm-modelplane
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: provider-helm-modelplane
  namespace: crossplane-system
---
# Apply the DRC to provider-helm automatically via ImageConfig.
apiVersion: pkg.crossplane.io/v1beta1
kind: ImageConfig
metadata:
  name: provider-helm-modelplane
spec:
  matchImages:
  - type: Prefix
    prefix: xpkg.upbound.io/upbound/provider-helm
  runtime:
    configRef:
      name: provider-helm-modelplane

  


      
    
  



## Install Modelplane

The Modelplane Configuration adds the Modelplane APIs and the composition
functions that reconcile them:


  
    
      configuration.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The Modelplane Crossplane Configuration. Installing it adds the Modelplane
# APIs (InferenceGateway, InferenceClass, InferenceCluster, ModelDeployment,
# ModelCache, ModelService) and the composition functions that reconcile them.
apiVersion: pkg.crossplane.io/v1
kind: Configuration
metadata:
  name: modelplane
spec:
  package: xpkg.upbound.io/modelplane/modelplane:v0.3.1

  



Wait until the configuration is healthy:

```bash
kubectl wait configuration/modelplane --for=condition=Healthy --timeout=5m
```

## Next step

The control plane is running but has nothing to schedule against yet. In the
next step, you'll [build the platform](/getting-started/build-the-platform/) to provision a GPU cluster and
publish what hardware it offers.


---

# Qwen3-8B

Source: /recipes/qwen3-8b/


An 8.2B dense chat model on a single NVIDIA L4. The smallest recipe: one
`Standalone` engine, no cache, weights pulled straight from Hugging Face.

This recipe was run end to end; the `InferenceClass` and `ModelDeployment` are
the exact manifests from that run. Apply the platform side first, then the ML
side.

## Validated deployments









  Dense
  8B
  16,384 ctx
  vLLM


  
    Cloud
    
      
        
          
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        BF16
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.23.0
    
  
  
  
  
    Features
    
      
        
          
          Speculative decoding
        
        for low latency and small batch sizes
      
    
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/qwen3-8b
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the L4 shape, validated serving Qwen3-8B on EKS.
#
# One NVIDIA L4 on an EKS g6.xlarge. The single GPU is a claim: DRA device;
# the scheduler matches a ModelDeployment's nodeSelector against its declared
# capacity and DRA binds it to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-l4-1x-g6
spec:
  description: "EKS g6.xlarge, 1x NVIDIA L4"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6.xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      # The L4's real usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 24GB.
      memory: { value: "23034Mi" }

  




  
    
      inference-cluster.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An EKS InferenceCluster with one L4 node pool, labeled for the
# ModelDeployment's clusterSelector to target.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-l4
  labels:
    modelplane.ai/region: us
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2
  nodePools:
  - name: gpu-l4
    className: eks-l4-1x-g6
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west-2a

  



## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen3-8B served on a single NVIDIA L4, validated end to end on EKS.
#
# An 8.2B dense model is a single Standalone engine: one self-contained vLLM
# pod, no ModelCache, weights pulled straight from Hugging Face. The flags carry
# real meaning beyond fit:
#
#   --tool-call-parser=hermes        the parser for Qwen3 dense (qwen3_xml is
#                                    for Qwen3-Coder, not this model). Qwen3's
#                                    tool-use template ships in the tokenizer,
#                                    so no --chat-template is needed.
#   --reasoning-parser=qwen3 with
#   --default-chat-template-kwargs   turns thinking off. Qwen3 thinks by
#                                    default, burying a one-line answer under a
#                                    <think> block and forbidding greedy decode.
#   --max-model-len / --gpu-memory-utilization  L4 fit, not correctness.
#
# No --port or --host: Modelplane's routing expects the engine on its default
# :8000 with a /health probe, and passes args through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us
      engines:
      - name: qwen3-8b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - "--model=Qwen/Qwen3-8B"
                - "--served-model-name=qwen"
                - "--max-model-len=16384"
                - "--gpu-memory-utilization=0.92"
                - "--reasoning-parser=qwen3"
                - "--default-chat-template-kwargs={\"enable_thinking\": false}"
                - "--enable-auto-tool-choice"
                - "--tool-call-parser=hermes"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the qwen3-8b deployment's endpoints as a single OpenAI-compatible URL.
# Modelplane labels each composed ModelEndpoint with the deployment name, so this
# selector reaches every replica. Read the public address from status.address:
#   kubectl get ms qwen3-8b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b

  



## Speculative decoding

The same model and platform also serve with n-gram (prompt-lookup) speculative
decoding, which proposes tokens by matching the prompt and so needs no draft
model or second set of weights. On copy-heavy output, editing a pasted code
block where most output tokens are copied from the prompt, it roughly doubles
decode throughput and halves the time per output token:

| Metric | Without speculation | With n-gram speculation |
|---|---|---|
| Output token throughput (tok/s) | 16.10 | 39.01 |
| Mean TPOT (ms/token) | 60.20 | 24.21 |

Measured on a single L4 (`vllm/vllm-openai:v0.23.0`, Qwen3-8B, 30 copy-heavy
prompts at concurrency 1) against the same model without `--speculative-config`;
the speculative run accepted 65% of drafted tokens, a mean acceptance length of
4.27 of 5. Speculation proposes several tokens per decode step and verifies them in
one forward pass, so when the output repeats the prompt most proposed tokens are
accepted at once, without changing what the model would have generated.

This variant was run end to end on GKE on the same single-L4 platform shape;
the `ModelDeployment` below is the exact manifest from that run, and the
numbers above are from the same run. Apply it instead of (or alongside) the
deployment above:


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen3-8B served on a single NVIDIA L4 by vLLM with n-gram (prompt-lookup)
# speculative decoding, validated end to end (the model layer is cloud-agnostic;
# the same manifest serves on EKS and GKE).
#
# n-gram speculation proposes the next tokens by matching a short suffix of what
# has been generated so far against earlier text in the prompt, then verifies the
# guess in one forward pass. It needs no draft model and no second set of weights,
# so it stays a single Standalone engine with no ModelCache. That is deliberate:
# Modelplane cannot yet stage a separate draft model on cache (modelplaneai/
# modelplane#281), so this is the speculative flavor that works today.
#
#   --speculative-config            method=ngram with num_speculative_tokens=5
#                                   proposes up to 5 tokens per step;
#                                   prompt_lookup_min/max=2..4 set the n-gram
#                                   suffix lengths matched against the prompt.
#                                   It pays off only when output repeats the
#                                   input - e.g. editing a pasted code block,
#                                   where most output tokens are copied verbatim.
#   --default-chat-template-kwargs  turns thinking off. Qwen3 thinks by default,
#                                   and a <think> block is novel text absent from
#                                   the prompt, so prompt-lookup cannot accelerate
#                                   it. Off, the output is mostly the copied code,
#                                   which is exactly what n-gram speeds up.
#   --max-model-len / --gpu-memory-utilization  L4 fit, not correctness. n-gram
#                                   adds only a small proposal buffer, no weights,
#                                   so the budget matches the plain Qwen3-8B recipe.
#
# No --port or --host: Modelplane's routing expects the engine on its default
# :8000 with a /health probe, and passes args through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-8b-spec
  namespace: ml-team
spec:
  # One replica, matched to any compatible InferenceCluster by device capacity.
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen3-8b-spec
        members:
    # A single self-contained vLLM pod. The container named "engine" is the
    # inference server; its image and args pass through verbatim.
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # An 8B model needs most of an L4. >=20Gi selects the L4 (which reports
          # ~23Gi) without over-constraining. DRA evaluates this CEL against the
          # InferenceClass device, then against the GPU's ResourceSlice on bind.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - "--model=Qwen/Qwen3-8B"
            # The id clients pass as "model" in OpenAI requests.
                - "--served-model-name=qwen3-8b-spec"
            # Cap the context so the KV cache fits beside the weights on the L4.
                - "--max-model-len=16384"
                - "--gpu-memory-utilization=0.92"
            # Enable n-gram speculative decoding (no draft model, no cache).
                - "--speculative-config={\"method\": \"ngram\", \"num_speculative_tokens\": 5, \"prompt_lookup_max\": 4, \"prompt_lookup_min\": 2}"
            # Thinking off, so output copies the prompt and prompt-lookup pays off.
                - "--default-chat-template-kwargs={\"enable_thinking\": false}"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the qwen3-8b-spec deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name, so
# this selector reaches every replica. Read the public address from
# status.address:
#   kubectl get ms qwen3-8b-spec -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-8b-spec
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b-spec

  



Speculation is active when the engine logs its `SpeculativeConfig` at startup
(`method='ngram'`). The call below pastes a code block and asks for a small edit,
the copy-heavy case n-gram accelerates, so most output tokens are matched straight
from the prompt:

```bash
ADDR=$(kubectl get ms qwen3-8b-spec -n ml-team -o jsonpath='{.status.address}')
curl -s "$ADDR/v1/chat/completions" -H 'Content-Type: application/json' -d '{
  "model": "qwen3-8b-spec",
  "messages": [{"role":"user","content":"Return this Python function unchanged except rename the variable `total` to `subtotal`. Output only the code.\n\ndef cart(items):\n    total = 0\n    for item in items:\n        total += item.price\n    return total"}],
  "max_tokens": 200, "temperature": 0 }'
```

With the engine running, its logs report how many proposed tokens it accepts:

```bash
kubectl logs -n ml-team -l modelplane.ai/deployment=qwen3-8b-spec \
  | grep "SpecDecoding metrics"
```



---

# Set Up the Gateway

Source: /platform/inference-gateway/

**API:** [`modelplane.ai/v1alpha1` · InferenceGateway](/reference/inferencegateways/)

The `InferenceGateway` sets up the control plane's front door: one unified,
OpenAI-compatible address that every `ModelService` is exposed through, routing
each request on to the inference cluster serving it.

The `InferenceGateway` is a singleton: create exactly one, named `default`, on
your Modelplane control plane. It fronts every inference cluster in the fleet, so
you don't create one per cluster.

The `backend` field selects which gateway runs it. `Traefik` is the only value
today.

On a cloud cluster with a native LoadBalancer controller, the gateway's `Service`
gets an external address on its own. On kind or bare-metal, where there's no such
controller, set `spec.traefik.loadBalancer: MetalLB` and give it an address pool
in `spec.traefik.metallb.addressPool` so the gateway gets an IP. See the example
below.

Once the gateway is ready, read its external address from `status.address`:

```bash
kubectl get ig default -o jsonpath='{.status.address}'
```

That address is the host of every `ModelService` URL
(`http://<address>/<namespace>/<service>`), so it's what you hand to ML teams.
## Example


  
    
      inference-gateway.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The InferenceGateway creates a unified, OpenAI-compatible endpoint on the
# control plane cluster. It installs Traefik Proxy and creates a Gateway that
# routes traffic to model replicas on remote inference clusters.
#
# Create one InferenceGateway per control plane. It must be named "default".
#
# For kind or bare-metal clusters, set loadBalancer to MetalLB and configure an
# address pool. For cloud clusters with native LoadBalancer support, omit the
# loadBalancer field entirely.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: default
spec:
  backend: Traefik
  traefik:
    version: "40.2.0"

    # Remove the loadBalancer section if your cluster supports LoadBalancer
    # services natively (e.g. GKE, EKS).
    loadBalancer: MetalLB
    metallb:
      addressPool: "172.18.255.200-172.18.255.250"

  





---

# Why Modelplane

Source: /overview/why/



Open-weight models are becoming the choice for organizations: they can be
post-trained, including with reinforcement learning, to compete with frontier
models, and they put cost, governance, and data sovereignty back under the
organization's control. As they do, platform teams are 
increasingly asked to provide GPU inference to their ML and development teams the 
same way they already provide cloud infrastructure.

## Kubernetes is becoming the default orchestrator

Kubernetes is rapidly becoming the default orchestrator for inference. The broader 
cloud-native community is investing heavily to make it a first-class platform for
AI workloads, adding device-aware scheduling, multi-node inference, distributed
serving, and accelerator management. The major open source inference projects are
converging on it; among them are vLLM, SGLang, NVIDIA Dynamo, llm-d, Ray, Slurm,
KubeAI, and Kueue. Neoclouds like Baseten and CoreWeave have standardized on
Kubernetes for their own operations. Inside a single cluster, the open source
stack is now strong.

## Inference is a fleet problem

Inference, however, almost always runs across more than one cluster. Accelerator
availability scatters capacity across hardware types, providers, and regions.
Sovereignty and compliance pin workloads to specific locations. Operators run
across multiple clouds and on-premise environments. Large clusters
concentrate failure and risk, so fleets of smaller clusters are often preferable,
and inference workloads don't bin-pack the way other workloads do.

Inference grows into a fleet, and a new set of problems appears above
any single cluster:

- Deciding where each model runs across available capacity.
- Optimizing placement across heterogeneous accelerators.
- Failing over across clouds and regions.
- Routing by cost, latency, and sovereignty requirements.
- Provisioning new capacity as demand grows.
- Caching and distributing model weights across the fleet.
- Managing the lifecycle of models, clusters, and infrastructure as one system.

Open source addresses pieces of this but none brings all the pieces together in a
fleet-wide system of record that manages placement, caching, capacity, policy, and
routing across an entire fleet. The labs, hyperscalers, and managed providers have
all solved these problems in a proprietary way, but the open equivalent does not
yet exist.

## Modelplane extends Kubernetes to manage the fleet

Modelplane does for the fleet what Kubernetes does for the cluster. It's the open
source control plane above your inference clusters across cloud, neocloud, and
on-premise: it places model deployments, autoscales replicas, provisions and
manages the infrastructure underneath, caches and distributes model weights, and
routes inference through one unified gateway with fallback to managed providers.
It turns "I need this model served" into a stable endpoint for any ML team.

Modelplane composes these projects rather than replacing them, and stays neutral
across models, accelerators, clouds, and serving stacks. It's built on
[Crossplane](https://crossplane.io) and extends Kubernetes to manage inference
at the fleet level. Modelplane is open source, Apache 2 licensed, and we plan to
donate it to a neutral open source foundation later this year.




  
    
    How Modelplane works
  
  The architecture, the resources, and what happens when you deploy a model.
  →



  
    
    FAQ
  
  How Modelplane compares to cluster orchestrators and managed providers, and what it requires.
  →









---

# Qwen2.5-7B

Source: /recipes/qwen2.5-7b/


A 7B dense chat model served from an AWQ INT4 quantization on a single NVIDIA
A16 on Vultr: one `Standalone` engine, no cache, weights pulled straight from
Hugging Face. The A16 slice on the `vcg-a16-6c-64g-16vram` plan carries 16 GiB
of VRAM, so the INT4 weights (~5 GiB) fit with headroom for KV cache;
`--gpu-memory-utilization=0.85` and `--enforce-eager` keep the engine inside
the small card.

This recipe was run end to end on Vultr (`ewr`); the `InferenceClass`,
`InferenceCluster`, and `ModelDeployment` are the exact manifests from that
run. GPU plans are region-gated on Vultr, so check the plan is offered in your
region before applying. Apply the platform side first, then the ML side.

## Validated deployments









  Dense
  7B
  8,192 ctx
  vLLM


  
    Cloud
    
      
        
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        AWQ INT4
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.9.2
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/qwen2.5-7b
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the A16 shape, validated serving Qwen2.5-7B on Vultr.
#
# One NVIDIA A16 on a Vultr vcg-a16-6c-64g-16vram plan: 16 GiB of VRAM is
# enough for a 7B AWQ INT4 model with KV cache headroom. GPU plans are
# region-gated on Vultr, so check the plan is offered in the cluster's region
# before applying.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: vultr-a16-1x
spec:
  description: "Vultr vcg-a16-6c-64g-16vram, 1x NVIDIA A16 16GB"
  provisioning:
    provider: Vultr
    vultr:
      plan: vcg-a16-6c-64g-16vram
      accelerator:
        type: nvidia-a16
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.6.0" }
    capacity:
      memory: { value: "16384Mi" }

  




  
    
      inference-cluster.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A Vultr Kubernetes Engine cluster in ewr (New Jersey) with a single A16 node
# offering the class above.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: vultr-a16
  labels:
    modelplane.ai/region: ewr
spec:
  cluster:
    source: Vultr
    vultr:
      region: ewr
  nodePools:
  - name: gpu-a16
    className: vultr-a16-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  



## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen2.5-7B served on a single NVIDIA A16, validated end to end on Vultr.
#
# A 7B AWQ INT4 model is a single Standalone engine: one self-contained vLLM
# pod, no ModelCache, weights pulled straight from Hugging Face. The 16 GiB
# card is the constraint: --gpu-memory-utilization=0.85 leaves headroom for
# the card's overhead, and --enforce-eager skips CUDA graph capture to save
# memory at some latency cost. It carries no clusterSelector, so device
# capacity alone matches it to any compatible GPU in the fleet.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-7b
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen-7b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("14Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.9.2
                args:
                - --model=Qwen/Qwen2.5-7B-Instruct-AWQ
                - --served-model-name=qwen-7b
                - --max-model-len=8192
                - --gpu-memory-utilization=0.85
                - --enforce-eager

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms qwen-7b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-7b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-7b

  





---

# Nemotron-3.5-Lightning

Source: /recipes/nemotron-3.5-lightning/


NVIDIA's Nemotron-3.5-Lightning, an open 30B mixture-of-experts model with 3B
active parameters built for the execution layer of long-running agents, served
NVFP4 as a single `Standalone` vLLM engine on one H100 node on Nebius.
The NVFP4 checkpoint (~20 GiB) fits a single GPU with headroom for the KV and
Mamba caches, so the engine needs no tensor parallelism, no gang, and no
prefill/decode disaggregation. Weights stage once to a `ModelCache` on a
Nebius shared filesystem and mount at `/mnt/models`.

This recipe was run end to end on Nebius (`eu-north`): serving and tool
calling validated on a single H100 node.
`nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4` is a public repository
(OpenMDW-1.1), so no Hugging Face token or Secret is needed. Apply the
platform side first, then the ML side.

## Validated deployments









  MoE
  30B A3B
  65,536 ctx
  vLLM


  
    Cloud
    
      
        
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        NVFP4
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.27.1
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/nemotron-3.5-lightning
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing a Nebius gpu-h100-sxm node with 1x NVIDIA
# H100 80GB. Nebius sizes nodes by platform + preset rather than an
# instance type; gpu-h100-sxm + 1gpu-16vcpu-200gb is one single-H100 SXM
# node. The devices block describes the hardware DRA-style; it is what
# the Nemotron ModelDeployment's nodeSelector matches on.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-1x
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # H100 80GB real usable VRAM (NVIDIA DRA driver), not nominal 80GB.
      memory: { value: "81559Mi" }

  




  
    
      inference-cluster-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceCluster backed by a Nebius mk8s cluster with a single 1x
# H100 GPU node group. Modelplane provisions the full mk8s cluster (VPC,
# control plane, system + GPU node groups) and installs the inference
# stack. ModelCache RWX storage is auto-provisioned via a Nebius shared
# filesystem, so model-cache.yaml works unchanged.
#
# Auth is the Nebius ClusterProviderConfig named default (service-account
# key + projectID); Modelplane reuses that identity to reach the cluster.
#
# eu-north1 (Finland) keeps inference in the EU for data residency.
#
# Clean teardown - delete the ML resources first (they hold a usage on the
# cluster), then the cluster:
#   kubectl delete modeldeployment,modelservice,modelcache nemotron-lightning -n ml-team
#   kubectl delete inferencecluster nebius-eu-north --cascade=foreground
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}

  nodePools:
  - name: gpu-h100
    className: nebius-h100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  



## Deployment


  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The model weights, staged once per cluster on a Nebius shared filesystem
# (RWX) and mounted at /mnt/models in the serving pod, so the engine reads
# the NVFP4 weights (~20 GiB) locally instead of pulling them from Hugging
# Face on every start.
#
# nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 is a public repo, so no
# authSecret / HF token is needed. Add one only if you point this at a gated
# repo.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: nemotron-lightning
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
    sizeGiB: 50

  




  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Nemotron-3.5-Lightning (30B total / 3B active MoE, hybrid
# Mamba-Transformer) served NVFP4 as a single Standalone vLLM engine on one
# H100 80GB, weights streamed from the shared ModelCache at /mnt/models. The
# NVFP4 checkpoint (~20 GiB) fits a single GPU with headroom for the KV and
# Mamba caches, so no tensor parallelism, no gang, and no prefill/decode
# disaggregation are needed.
#
# Notes on the engine flags:
#   Quantization needs no flag - the checkpoint auto-detects as
#     modelopt_mixed.
#   --moe-backend=humming and --linear-backend=humming are the cookbook's
#     base-configuration kernels.
#   The --mamba-* flags configure the hybrid model's state-space cache: the
#     flashinfer backend with an FP16 SSM cache, stochastic rounding, and
#     the cookbook's align mode and horizontal SSU algorithm.
#   --reasoning-parser=nemotron_v3 extracts the thinking block;
#     --tool-call-parser=qwen3_coder is the parser Nemotron ships with, and
#     --enable-auto-tool-choice turns on server-side tool selection.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: nemotron-lightning
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: nemotron-lightning
      engines:
      - name: nemotron-lightning
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("79Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.27.1
                command: ["vllm", "serve", "/mnt/models"]
                args:
                - --served-model-name=nemotron-3.5-lightning
                - --moe-backend=humming
                - --linear-backend=humming
                - --max-num-seqs=256
                - --max-model-len=65536
                - --max-num-batched-tokens=32768
                - --enable-prefix-caching
                - --async-scheduling
                - --mamba-backend=flashinfer
                - --mamba-ssm-cache-dtype=float16
                - --enable-mamba-cache-stochastic-rounding
                - --mamba-cache-philox-rounds=5
                - --mamba-cache-mode=align
                - --mamba-ssu-algorithm=horizontal
                - --reasoning-parser=nemotron_v3
                - --enable-auto-tool-choice
                - --tool-call-parser=qwen3_coder

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms nemotron-lightning -n ml-team -o jsonpath='{.status.address}'
# then call it, e.g.:
#   curl "$ADDR/v1/chat/completions" -H 'Content-Type: application/json' \
#     -d '{"model":"nemotron-3.5-lightning","messages":[{"role":"user","content":"hello"}]}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: nemotron-lightning
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: nemotron-lightning

  





---

# Build the platform

Source: /getting-started/build-the-platform/

This is the platform team's side of Modelplane. You set up the gateway that
fronts your models, give the control plane cloud credentials, and register your
first GPU cluster: a hardware profile published as an `InferenceClass` and an
`InferenceCluster` that offers it.

In the next step, the ML team will create a model deployment that schedules
against this capacity without knowing which cluster it runs on.

## Prerequisites










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  
  
  
  
    Vultr
    
    
  
  


  







<ul>
<li>An AWS account with permissions to create EKS clusters, VPCs, and IAM roles
<li>AWS access key ID and secret access key










<ul>
<li>
<p>A GCP service account JSON key, granted these roles on the project:
<table>
	<thead>
			<tr>
					<th>Role
					<th>Needed for
			
	
	<tbody>
			<tr>
					<td><code>roles/container.admin
					<td>the cluster and its node pools
			
			<tr>
					<td><code>roles/compute.admin
					<td>the VPC network and subnet
			
			<tr>
					<td><code>roles/serviceusage.serviceUsageAdmin
					<td>enabling the APIs the cluster needs
			
			<tr>
					<td><code>roles/iam.serviceAccountAdmin
					<td>the node service account
			
			<tr>
					<td><code>roles/iam.serviceAccountKeyAdmin
					<td>the node service account’s key
			
			<tr>
					<td><code>roles/iam.serviceAccountUser
					<td>attaching that account to the nodes
			
			<tr>
					<td><code>roles/resourcemanager.projectIamAdmin
					<td>granting the node account <code>container.admin
			
	

<p>The last one is worth a look before you hand the key over. Modelplane grants
the node service account <code>roles/container.admin, so the credential doing the
provisioning has to be able to set project IAM policy.











<ul>
<li>An Azure account with permissions to create AKS clusters and managed identities
<li>An Azure service principal JSON with <code>clientId, <code>clientSecret, <code>subscriptionId, and <code>tenantId










<ul>
<li>A Nebius account with permissions to create clusters
<li>A Nebius service account JSON key and your project ID










<ul>
<li>A Vultr account with access to GPU plans
<li>A Vultr API key







## Set up the InferenceGateway


The `InferenceGateway` installs Traefik Proxy and MetalLB on the control plane.
Traefik routes inference traffic to model replicas. MetalLB assigns Traefik's
`LoadBalancer` service an external IP on kind, which doesn't have a cloud load
balancer. You need one named `default` per control plane.


If you run the control plane on a cloud cluster with native `LoadBalancer`
support, omit the `loadBalancer` field.


  
    
      inference-gateway.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The InferenceGateway creates a unified, OpenAI-compatible endpoint on the
# control plane cluster. It installs Traefik Proxy and creates a Gateway that
# routes traffic to model replicas on remote inference clusters.
#
# Create one InferenceGateway per control plane. It must be named "default".
#
# For kind or bare-metal clusters, set loadBalancer to MetalLB and configure an
# address pool. For cloud clusters with native LoadBalancer support, omit the
# loadBalancer field entirely.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceGateway
metadata:
  name: default
spec:
  backend: Traefik
  traefik:
    version: "40.2.0"

    # Remove the loadBalancer section if your cluster supports LoadBalancer
    # services natively (e.g. GKE, EKS).
    loadBalancer: MetalLB
    metallb:
      addressPool: "172.18.255.200-172.18.255.250"

  



Wait until the gateway is ready:

```bash
kubectl wait --for=condition=Ready ig/default --timeout=5m
```

## Configure cloud credentials

Give the control plane credentials so it can provision clusters in your cloud
account.










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  
  
  
  
    Vultr
    
    
  
  


  







<p>Create an AWS credentials file:

  
    ini
    
      
        
        
      
    
  
  [default]
aws_access_key_id = 
aws_secret_access_key = 

<p>Create a Kubernetes secret:

  
    bash
    
      
        
        
      
    
  
  kubectl create secret generic aws-creds \
  --from-file=credentials= \
  -n crossplane-system

<p>Apply the <code>ClusterProviderConfig referencing your secret:

  
    
      clusterproviderconfig-aws.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Points the AWS provider at the credentials Secret you created. Named default,
# so InferenceClusters with an EKS source use it without further configuration.
apiVersion: aws.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: credentials

  











<p>Create a Kubernetes secret:

  
    bash
    
      
        
        
      
    
  
  kubectl create secret generic gcp-creds \
  --from-file=credentials=.json \
  -n crossplane-system

<p>Apply the <code>ClusterProviderConfig, setting <code>projectID to your GCP project:

  
    
      clusterproviderconfig-gke.yaml
      
          
          
        
      
    
    # Points the GCP provider at the credentials Secret you created. Named default,
# so InferenceClusters with a GKE source use it without further configuration.
apiVersion: gcp.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  projectID: my-gcp-project  # replace with your GCP project
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-creds
      key: credentials

  


  
    bash
    
      
        
        
      
    
  
  curl -fsSL /examples/getting-started/clusterproviderconfig-gke.yaml \
  | sed 's/my-gcp-project//' \
  | kubectl apply -f -











<p>Create a Kubernetes secret from your service principal JSON:

  
    bash
    
      
        
        
      
    
  
  kubectl create secret generic azure-credentials \
  --from-file=credentials.json=.json \
  -n crossplane-system

<p>Apply the <code>ClusterProviderConfig referencing your secret:

  
    
      clusterproviderconfig-azure.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Points the Azure providers at the credentials Secret you created. Named
# default, so InferenceClusters with an AKS source use it without further
# configuration. The Secret carries a service principal JSON with
# clientId, clientSecret, subscriptionId, and tenantId.
apiVersion: azure.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: azure-credentials
      key: credentials.json

  











<p>Create a Kubernetes secret from your service account JSON:

  
    bash
    
      
        
        
      
    
  
  kubectl create secret generic nebius-credentials \
  --from-file=credentials.json=.json \
  -n crossplane-system

<p>Apply the <code>ClusterProviderConfig, setting <code>projectID to your Nebius project:

  
    
      clusterproviderconfig-nebius.yaml
      
          
          
        
      
    
    # Points the Nebius provider at the credentials Secret you created. Named
# default, so InferenceClusters with a Nebius source use it without further
# configuration - Modelplane also reuses its credentials Secret to
# authenticate to the clusters it provisions.
apiVersion: nebius.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  identity:
    type: ServiceAccount
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: nebius-credentials
      key: credentials.json
  projectID: project-e00example

  


  
    bash
    
      
        
        
      
    
  
  curl -fsSL /examples/getting-started/clusterproviderconfig-nebius.yaml \
  | sed 's/project-e00example//' \
  | kubectl apply -f -











<p>Create a Kubernetes secret from your API key:

  
    bash
    
      
        
        
      
    
  
  kubectl create secret generic vultr-credentials \
  --from-literal=api-key= \
  -n crossplane-system

<p>Apply the <code>ClusterProviderConfig referencing your secret:

  
    
      clusterproviderconfig-vultr.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Points the Vultr provider at the API key Secret you created. Named
# default, so InferenceClusters with a Vultr source use it without further
# configuration. The API key only provisions clusters - VKE kubeconfigs
# embed static client certificates, so consumers never need it to reach
# the clusters Modelplane provisions.
apiVersion: vultr.m.upbound.io/v1beta1
kind: ClusterProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: vultr-credentials
      key: api-key

  







## Publish hardware and register the cluster

The `InferenceClass` describes a hardware profile and how to provision it. The
`InferenceCluster` registers a cluster that offers it. Apply both:










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  
  
  
  
    Vultr
    
    
  
  


  








  
    
      platform.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l4-1x-g6
spec:
  description: "EKS g6.xlarge, 1x NVIDIA L4"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6.xlarge
      diskSizeGb: 50
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "23034Mi" }   # L4's real reported VRAM (not the nominal 24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: EKS
    eks:
      region: us-east-1
  nodePools:
  - name: gpu-l4
    className: l4-1x-g6
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-east-1b

  

<p>Modelplane provisions the cluster. This takes about 15 minutes:


  
    bash
    
      
        
        
      
    
  
  kubectl wait --for=condition=Ready ic/eks-us-east --timeout=20m










<p>Apply the manifest:

  
    
      platform.yaml
      
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-l4-1x-g2
spec:
  description: "GKE g2-standard-8, 1x NVIDIA L4"
  provisioning:
    provider: GKE
    gke:
      machineType: g2-standard-8
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "23034Mi" }   # L4's real reported VRAM (not the nominal 24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: starter
  labels:
    modelplane.ai/region: us-central
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
  - name: gpu-l4
    className: gke-l4-1x-g2
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-central1-a

  



  
    bash
    
      
        
        
      
    
  
  kubectl apply -f /examples/getting-started/gke/platform.yaml


<p>Modelplane provisions the cluster. This takes about 15 minutes:


  
    bash
    
      
        
        
      
    
  
  kubectl wait --for=condition=Ready ic/starter --timeout=20m













  
    
      platform.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a10-1x
spec:
  description: "AKS Standard_NV36ads_A10_v5, 1x NVIDIA A10 24GB"
  provisioning:
    provider: AKS
    aks:
      # verify: Azure has no L4 SKU; NVadsA10v5 is the small-GPU option (>=20Gi).
      # Confirm it runs the DRA / GPU-operator path in your subscription.
      vmSize: Standard_NV36ads_A10_v5
      diskSizeGb: 100
      accelerator:
        type: nvidia-a10
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.6.0" }
    capacity:
      memory: { value: "24564Mi" }   # verify: A10 reported VRAM (~24GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-westeurope
  labels:
    modelplane.ai/region: westeurope
spec:
  cluster:
    source: AKS
    aks:
      location: westeurope
  nodePools:
  - name: gpua10
    className: a10-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  

<p>Modelplane provisions the cluster. This takes about 15 minutes:


  
    bash
    
      
        
        
      
    
  
  kubectl wait --for=condition=Ready ic/aks-westeurope --timeout=20m











  
    
      platform.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x
spec:
  description: "Nebius gpu-l40s-a, 1x NVIDIA L40S 48GB"
  provisioning:
    provider: Nebius
    nebius:
      # gpu-l40s-a (L40S PCIe, Intel Ice Lake) is available in eu-north1.
      platform: gpu-l40s-a
      preset: 1gpu-8vcpu-32gb
      diskSizeGb: 100
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
      cudaComputeCapability: { version: "8.9.0" }
    capacity:
      memory: { value: "46068Mi" }   # L40S reported VRAM (48GB nominal)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-l40s
    className: l40s-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  

<p>Modelplane provisions the cluster. This takes about 15 minutes:


  
    bash
    
      
        
        
      
    
  
  kubectl wait --for=condition=Ready ic/nebius-eu-north --timeout=20m











  
    
      platform.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x
spec:
  # The largest GPU plan available in ewr. Vultr GPU plans are region-gated;
  # list what a region offers with the /v2/regions/<region>/availability API
  # before picking another plan.
  description: "Vultr vcg-l40s-16c-180g-48vram, 1x NVIDIA L40S 48GB"
  provisioning:
    provider: Vultr
    vultr:
      plan: vcg-l40s-16c-180g-48vram
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
      cudaComputeCapability: { version: "8.9.0" }
    capacity:
      memory: { value: "46068Mi" }   # L40S's real reported VRAM (not the nominal 48GB)
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: vultr-ewr
  labels:
    modelplane.ai/region: ewr
spec:
  cluster:
    source: Vultr
    vultr:
      region: ewr
  nodePools:
  - name: gpu-l40s
    className: l40s-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  

<p>Modelplane provisions the cluster. This takes about 15 minutes:


  
    bash
    
      
        
        
      
    
  
  kubectl wait --for=condition=Ready ic/vultr-ewr --timeout=20m









  
    
    Note
  
  
    <p>Modelplane is reconciling the infrastructure against the source of truth, the
manifest you just applied.
<p>While you wait, Modelplane is creating the cloud cluster and its GPU node
pool, then installing the inference stack with LeaderWorkerSet for multi-node
serving, llm-d for inference-aware routing, Envoy Gateway for traffic
management, and the storage class for model weights. This is the same reconciliation loop Crossplane uses to configure other
infrastructure, extended to the inference layer.

  



Once the cluster is `Ready` the ML team can deploy a model on it.




  
    
    Note
  
  
    A cloud GPU cluster costs money while it runs. To stop the tour and resume
later, follow Clean up.
  



## Next step

Now that the platform is provisioned, the ML team can [deploy a model](/getting-started/deploying-a-model/) by describing what the model needs, not the infrastructure.


---

# Collecting engine metrics

Source: /guides/collecting-engine-metrics/


Scraping an inference engine's Prometheus metrics, shown on the smallest serving
shape: a 0.5B Qwen chat model on one NVIDIA L4. vLLM publishes metrics at
`/metrics` on its serving port with no extra flag, and Modelplane runs a
Prometheus on every workload cluster with `PodMonitor` discovery open across
namespaces, so scraping the engine is a `PodMonitor` plus a `port-forward`. The
model is only the subject; the same wiring fits any engine, with the SGLang,
leader/worker, and prefill/decode differences noted at the end.

This was run end to end on GKE. The `InferenceClass` and `ModelDeployment` are the
exact manifests from that run, and the `PodMonitor` below scraped this deployment.
Apply the platform side first, then the ML side.

## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the L4 shape on GKE, validated serving Qwen2.5-0.5B.
#
# One NVIDIA L4 on a g2-standard-8. The GPU is declared as a DRA device: the
# scheduler matches a ModelDeployment's nodeSelector against this capacity, then
# DRA binds the physical GPU to the serving pod. A 0.5B model uses a sliver of
# this L4 - the shape is shared with the larger L4 examples, not sized for it.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-l4-1x-g2
spec:
  description: "GKE g2-standard-8, 1x NVIDIA L4"
  provisioning:
    provider: GKE
    gke:
      machineType: g2-standard-8
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      # The L4's real usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 24GB.
      memory: { value: "23034Mi" }

  




  
    
      inference-cluster.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # GKE InferenceCluster with one L4 node pool. Replace the project ID before
# applying. No clusterSelector targets it; the ModelDeployment matches on device
# capacity alone, so it lands here or on any other compatible cluster.
#
# Modelplane installs an in-cluster Prometheus on this cluster (the monitoring
# namespace) with open PodMonitor discovery - the metrics section of the example
# scrapes the engine through it.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-l4-single
  labels:
    modelplane.ai/cloud: gke
    modelplane.ai/region: us-central
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
  - name: gpu-l4
    className: gke-l4-1x-g2
    nodeCount: 1
    zones:
    - us-central1-a
    minNodeCount: 0
    maxNodeCount: 4

  



## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen2.5-0.5B-Instruct served on a single NVIDIA L4 by vLLM, validated end to
# end on GKE.
#
# A 0.5B dense model is the smallest useful serving shape: one Standalone vLLM
# pod, no ModelCache, weights pulled straight from Hugging Face. It barely touches
# the L4's VRAM, so the flags here are about behavior, not fit:
#
#   --max-model-len=16384   caps the context; the default 32K KV cache is wasteful
#                           for a model this size and a demo this small.
#   --served-model-name     the id clients pass as "model" in OpenAI requests.
#
# vLLM exposes Prometheus metrics at /metrics on its serving port (:8000) with no
# extra flag, which is what the example's PodMonitor scrapes.
#
# No --port or --host: Modelplane's routing expects the engine on its default
# :8000 with a /health probe, and passes args through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen2-5-0-5b
  namespace: ml-team
spec:
  # One replica, matched to any compatible InferenceCluster by device capacity.
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen2-5-0-5b
        members:
    # A single self-contained vLLM pod. The container named "engine" is the
    # inference server; its image and args pass through verbatim.
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # The model needs almost no VRAM; >=20Gi simply pins it to the L4 pool
          # this example provisions (the L4 reports ~23Gi). DRA evaluates this CEL
          # against the InferenceClass device, then against the GPU's
          # ResourceSlice when it binds the claim.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - "--model=Qwen/Qwen2.5-0.5B-Instruct"
                - "--served-model-name=qwen2.5-0.5b"
                - "--max-model-len=16384"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the qwen2-5-0-5b deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name, so
# this selector reaches every replica. Read the public address from status.address:
#   kubectl get ms qwen2-5-0-5b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen2-5-0-5b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen2-5-0-5b

  



## Scraping the metrics

The `PodMonitor` selects engine pods by the `modelplane.ai/serving` label
Modelplane stamps on them, and the `monitoring` namespace Prometheus discovers any
`PodMonitor`, so this is the whole config. The engine container port is unnamed,
so reference it by number with `targetPort`:


  
    
      podmonitor.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: qwen2-5-0-5b-metrics
  namespace: default
spec:
  selector:
    matchExpressions:
    - key: modelplane.ai/serving   # carried by every serving pod
      operator: Exists
  podMetricsEndpoints:
  - targetPort: 8000
    path: /metrics
    interval: 30s

  



The engine pods and the `PodMonitor` CRD live on the workload cluster, not the
control plane, so apply it there. Then read the metrics from the in-cluster
Prometheus over a `port-forward`:

```bash
kubectl -n monitoring port-forward svc/prometheus-prometheus 9090:9090   # workload cluster
# open http://localhost:9090, Status > Targets to confirm the scrape, then query
# e.g. vllm:num_requests_running or vllm:gpu_cache_usage_perc
```

### Other engine shapes

The `PodMonitor` above fits a single-pod vLLM engine. The selector and port shift
by shape:

- **SGLang**: exposes `/metrics` only when the engine runs with
  `--enable-metrics`; otherwise it's identical (same selector, `targetPort: 8000`).
- **Leader/worker**: only the leader serves the API and carries
  `modelplane.ai/serving`, so the selector above already scrapes the leader alone;
  the workers expose nothing.
- **prefill/decode**: two engines, labelled `llm-d.ai/role: prefill` and
  `llm-d.ai/role: decode`. The prefill engine serves on `8000`; the decode engine
  sits behind the routing sidecar that takes `8000` and listens on `8001`, so
  scrape decode with `targetPort: 8001`. Select each by its role label to keep
  them apart.



---

# Define Hardware Classes

Source: /platform/inference-class/

**API:** [`modelplane.ai/v1alpha1` · InferenceClass](/reference/inferenceclasses/)


An `InferenceClass` is a tested recipe for a GPU node pool. It bundles:


- **Devices**: the node's hardware as a list of Dynamic Resource Allocation (DRA)
  style devices, each with a driver, count, typed attributes, and capacity. The
  scheduler matches a member's `nodeSelector` against these devices, and GPUs
  bind to pods through DRA.
- **Provisioning** (optional): how to create a node pool of this class on a
  specific cloud. Classes without provisioning are for existing clusters where
  the pool already exists.

Different clouds and GPU types imply different classes. A GKE L4 pool is
`gke-l4-1x-g2`. A bare-metal H100 pool is `h100-8x-byo` (no provisioning).

## Describing devices

A class's `devices` follow Kubernetes
[Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
(DRA), the mechanism modern Kubernetes uses to match GPUs to pods. Each device
has a `driver` (the vendor that owns it, such as `gpu.nvidia.com`), a `count`
(how many a node has), typed `attributes` (such as `architecture`), and
`capacity` (quantities, such as `memory`). This mirrors the shape the GPU's DRA
driver publishes on a real node, so what you declare here is what an ML team's
`nodeSelector` matches against and what DRA binds at runtime.

You author the attribute and capacity keys, and there's no fixed list. Pick the
ones an ML team would reasonably select on, the GPU memory, the architecture, the
compute capability, using the same names the driver reports.

## DRA and synthetic devices

Each device sets a `claim` discriminator:

- **`DRA`** (the default) is hardware a real DRA driver exposes, today GPUs.
  Modelplane both schedules against it and binds it to pods.
- **`Synthetic`** is described for scheduling only, never claimed. Use it for
  hardware that matters for placement but has no DRA driver yet, like an
  InfiniBand fabric.

## The device contract

The `driver`, attribute keys, and capacity keys a class declares are a contract
with the ML team: a `ModelDeployment`'s `nodeSelector` matches a pool only if the
class publishes the attributes and capacity it asks for. ML teams write those
matches as [CEL](https://cel.dev/) selectors over the keys you publish here. For
GPUs, these keys should mirror what the DRA driver reports, so the same selector
that places a deployment on the pool also binds the right device.

Publish a device's real usable capacity, not its nominal spec. An `80GB` H100
reports about `81559Mi` of usable memory, so a class that declares `80Gi` would
let a `nodeSelector` asking for `>= 80Gi` match the pool but then fail to bind the
GPU.

## Examples










  
  
  
  
    GKE L4
    
    
  
  
  
  
  
    EKS L4
    
    
  
  
  
  
  
    AKS H100
    
    
  
  
  
  
  
    Nebius H100
    
    
  
  
  
  
  
    Vultr L40S
    
    
  
  
  
  
  
    H100 bare-metal
    
    
  
  


  








  
    
      inference-class-gke-l4.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing GKE g2-standard-8 with one NVIDIA L4 GPU.
#
# The provisioning block tells Modelplane how to create a node pool of
# this class on GKE. The devices block describes what hardware a node of
# this class has, DRA-style - used by the scheduler to match models to
# clusters, and to form DRA ResourceClaims for claim: DRA devices.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-l4-1x-g2
spec:
  description: "GKE g2-standard-8, 1x NVIDIA L4"
  provisioning:
    provider: GKE
    gke:
      machineType: g2-standard-8
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "23034Mi" }

  











  
    
      inference-class-eks-l4.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing EKS g6.xlarge with one NVIDIA L4 GPU.
#
# The provisioning block tells Modelplane how to create a node group of
# this class on EKS. The devices block describes what hardware a node of
# this class has, DRA-style - used by the scheduler to match models to
# clusters, and to form DRA ResourceClaims for claim: DRA devices.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-l4-1x-g6
spec:
  description: "EKS g6.xlarge, 1x NVIDIA L4"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6.xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      # The L4's real usable VRAM, what the NVIDIA DRA driver reports in the
      # node's ResourceSlice, not its nominal 24GB. The scheduler matches a
      # nodeSelector against this, so declaring the marketing number would let
      # it place a model that wants 23Gi onto a node DRA then can't satisfy.
      memory: { value: "23034Mi" }

  











  
    
      inference-class-aks-h100.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing AKS Standard_ND96isr_H100_v5 with 8x
# NVIDIA H100.
#
# The provisioning block tells Modelplane how to create a node pool of
# this class on AKS. Azure sizes nodes by VM size, which determines the
# GPU model and count; the accelerator block is informational. The
# devices block describes what hardware a node of this class has,
# DRA-style - used by the scheduler to match models to clusters, and to
# form DRA ResourceClaims for claim: DRA devices.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: aks-h100-8x
spec:
  description: "AKS Standard_ND96isr_H100_v5, 8x NVIDIA H100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_ND96isr_H100_v5
      diskSizeGb: 200
      accelerator:
        type: nvidia-h100
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100 80GB's real usable VRAM, what the NVIDIA DRA driver reports,
      # not its nominal 80GB. A nodeSelector asking for >= 80Gi would never bind.
      memory: { value: "81559Mi" }

  











  
    
      inference-class-nebius-h100.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing Nebius gpu-h100-sxm with 8x NVIDIA H100.
#
# The provisioning block tells Modelplane how to create a node group of
# this class on Nebius mk8s. Nebius sizes nodes by platform + preset
# rather than an instance type; the preset determines the GPU, vCPU,
# and memory shape of each node. The devices block describes what
# hardware a node of this class has, DRA-style - used by the scheduler
# to match models to clusters, and to form DRA ResourceClaims for
# claim: DRA devices.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-8x
spec:
  description: "Nebius gpu-h100-sxm, 8x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 8gpu-128vcpu-1600gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100 80GB's real usable VRAM, what the NVIDIA DRA driver reports,
      # not its nominal 80GB. A nodeSelector asking for >= 80Gi would never bind.
      memory: { value: "81559Mi" }

  











  
    
      inference-class-vultr-l40s.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing Vultr's vcg-l40s-16c-180g-48vram plan with
# 1x NVIDIA L40S.
#
# The provisioning block tells Modelplane how to create a node pool of
# this class on VKE. Vultr sizes nodes by plan; the plan determines the
# GPU, vCPU, and memory shape of each node. The devices block describes
# what hardware a node of this class has, DRA-style - used by the
# scheduler to match models to clusters, and to form DRA ResourceClaims
# for claim: DRA devices.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: vultr-l40s-1x
spec:
  description: "Vultr vcg-l40s-16c-180g-48vram, 1x NVIDIA L40S 48GB"
  provisioning:
    provider: Vultr
    vultr:
      # List the plans available in a region via the Vultr API:
      #   curl -s "https://api.vultr.com/v2/regions/ewr/availability" \
      #     -H "Authorization: Bearer $VULTR_API_KEY" | jq '.available_plans[]'
      plan: vcg-l40s-16c-180g-48vram
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
      cudaComputeCapability: { version: "8.9.0" }
    capacity:
      # The L40S's real usable VRAM, what the NVIDIA DRA driver reports,
      # not its nominal 48GB. A nodeSelector asking for >= 48Gi would never bind.
      memory: { value: "46068Mi" }

  











  
    
      inference-class-h100-byo.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing a BYO 8x H100 node pool.
#
# No provisioning block: this class describes hardware that already
# exists on a bring-your-own cluster. Modelplane copies the devices block
# onto the cluster's status.gpuPools for the scheduler to match against.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: h100-8x-byo
spec:
  description: "BYO 8x NVIDIA H100 80GB"
  # DRA-style devices. These are the contract a ModelDeployment's
  # nodeSelector matches against. Keys are bare names; the domain comes
  # from each device's driver. claim: DRA devices are emitted as requests
  # in a DRA ResourceClaim; claim: Synthetic devices (here the InfiniBand
  # fabric, which has no DRA driver) are matched for scheduling only.
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100 80GB's real usable VRAM, what the NVIDIA DRA driver reports,
      # not its nominal 80GB. A nodeSelector asking for >= 80Gi would never bind.
      memory: { value: "81559Mi" }
  - name: nic
    claim: Synthetic
    driver: nic.nvidia.com
    count: 8
    attributes:
      linkType: { string: infiniband }

  









---

# Expose a Model

Source: /models/model-service/

**API:** [`modelplane.ai/v1alpha1` · ModelService](/reference/modelservices/)

A [`ModelDeployment`](/models/model-deployment/) serves a model, but its
replicas are scattered across the fleet with no single address. A `ModelService`
gives them one: a stable, unified, OpenAI-compatible URL that load-balances
across every replica, wherever it runs.

A service selects what to route to by label. Behind the scenes, Modelplane
creates one `ModelEndpoint`, a single reachable backend, for each replica of a
deployment and labels it. Two of those labels carry routing intent:

- `modelplane.ai/deployment`: the deployment the replica belongs to.
- `modelplane.ai/cluster`: the cluster the replica runs on.

Modelplane creates an endpoint only once its replica is Ready, serving and
reachable, and withdraws it if the replica later goes unhealthy. A service only
ever routes to replicas that can actually answer, so a deployment that's still
starting or scaling up has fewer endpoints behind its URL until those replicas
come up. You don't create endpoints yourself. You point a service at them.

`spec.endpoints` is a list, and the entries combine: the service routes to every
endpoint that any entry matches. The patterns below build on that.

## Route to a whole deployment

The common case: one selector matching a deployment's name reaches every replica,
wherever in the fleet they run.

```yaml {nocopy=true}
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b   # every replica of this deployment
```

## Route to part of a deployment

Add a second label to narrow within a deployment. A selector matches an endpoint
only when all its labels match, so pairing the deployment with a cluster routes to
just that cluster's replicas. This is how you take a cluster out of service
without redeploying: point the service at the clusters you want and leave one out,
and traffic drains to the rest.

```yaml {nocopy=true}
spec:
  endpoints:
  # Only the replicas on prod-us-east, e.g. while draining another cluster.
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b
        modelplane.ai/cluster: prod-us-east
```

## Route across several deployments

Give more than one entry to front several deployments behind the same URL. Each
entry contributes its matched endpoints. By default every entry carries equal
weight, so traffic splits evenly between entries and then spreads as evenly as
possible across the endpoints each one matches.

```yaml {nocopy=true}
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b-v2
```

## Split traffic by weight

Set a `weight` on an entry to give it a fixed share of traffic instead of an
equal one. Weights are relative: an entry weighted 80 next to one weighted 20
takes 80% of requests. The weight applies to the entry as a whole and spreads
as evenly as possible across the endpoints it matches, so scaling a deployment
up or down doesn't change its share. An entry without a `weight` defaults to 1.

This is the shape of a canary rollout: send most traffic to the stable deployment
and a sliver to the new one, then shift the ratio as confidence grows.

```yaml {nocopy=true}
spec:
  endpoints:
  - weight: 95
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b
  - weight: 5
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b-v2
```

The entries don't have to be deployments. One can select a manually created
[ModelEndpoint](/models/model-endpoint/) that points at an external
provider, so a service can send overflow or break-glass traffic to a SaaS
endpoint alongside your own replicas:

```yaml {nocopy=true}
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: kimi-k2
  - selector:
      matchLabels:
        modelplane.ai/external-provider: together
```

Endpoints with different path layouts coexist behind the one URL.

## Sending a request

The service's public address is on `status.address`, in the form
`http://<gateway>/<namespace>/<service-name>`:

```bash
ADDRESS=$(kubectl get ms qwen -n ml-team -o jsonpath='{.status.address}')
```

Append the OpenAI path and send a request. The `model` field is the name the
engine serves (its `--served-model-name`, or the model's Hugging Face id if you
didn't set one):

```bash
curl "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

## Alternate APIs

We call the endpoint OpenAI-compatible because the engines are, not because
Modelplane imposes it. The route matches the `/<namespace>/<service>/` prefix and
preserves the path below it on the way to the engine, so any API the engine serves
is reachable on the same URL.

Take a vLLM replica that also serves the Anthropic Messages API. It answers on
`.../v1/messages`, so a client that speaks it (including Claude Code, via
`ANTHROPIC_BASE_URL`) talks to it directly. The engine's operational paths come
through the same way: `.../health` and the Prometheus `.../metrics` are reachable
on the service URL.

There's one exception, and it's set by the deployment rather than the service.
[Disaggregated serving](/models/model-deployment/#disaggregated-serving)
reads OpenAI-format request bodies to pick a prefill and decode worker, so a
request in another API shape still reaches the engine but skips that
cache-aware routing. Unified serving forwards every API shape the same way.

## Example


  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelService exposes one or more ModelDeployments via a single
# OpenAI-compatible endpoint. It composes a Gateway-API HTTPRoute on the
# control plane that load-balances across every ModelEndpoint matching
# its selector.
#
# Modelplane composes one ModelEndpoint per ModelReplica, labeled
# `modelplane.ai/deployment: <deployment-name>`. So a ModelService with
# that label selector reaches every replica of the named deployment.
#
# Once the service is ready, its public address is on status.address:
#   kubectl get ms qwen3-8b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-8b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-8b

  





---

# How it schedules

Source: /architecture/scheduling/

**API:** [`modelplane.ai/v1alpha1` · ModelDeployment](/reference/modeldeployments/)

When an ML team creates a [ModelDeployment](/models/model-deployment/),
the fleet scheduler decides which cluster each replica runs on and which node
pool each engine uses. Platform teams don't drive it directly, but what they
publish, the clusters, their labels, and each pool's
[InferenceClass](/platform/inference-class/), is exactly what the
scheduler matches against. This page explains how it places work and where it
deliberately stops short, so you can reason about why a deployment landed where it
did.

## A pure function of observed state

The scheduler recomputes the whole placement from scratch on every reconcile. It
reads the deployment, every `InferenceCluster` with its published capacity, and
every existing `ModelReplica`, and returns a placement. Given the same inputs it
returns the same placement, so it's safe to run continuously.

The key consequence is stability. Existing replicas are *inputs*, not decisions.
A healthy replica is never moved to improve the global picture, even if a better
cluster appears later. This keeps placement from churning underneath a running
deployment.

## Two-level matching

The scheduler picks a `(cluster, pool)` for each replica in two stages, matching
against what the platform team published.

1. **Clusters** are filtered by `clusterSelector.matchLabels` against the
   standard Kubernetes labels on each `InferenceCluster`: tier, region, provider,
   compliance posture. This is organizational metadata, so string equality is
   enough. An unset selector matches every cluster.
2. **Pools** are filtered by matching each device request in a member's
   `nodeSelector.devices` against the devices a pool's `InferenceClass` publishes.
   A request is a real DRA request: a `count` and CEL selectors over a device's
   attributes and capacity, such as "a GPU with at least 141Gi of memory." A pool
   fits a member when it has devices satisfying every request, with `count` to
   cover them.

The CEL is the same expression an ML engineer would write in a DRA
`ResourceClaim`, evaluated against the devices the `InferenceClass` declares. The
keys a platform team puts on a class are the contract: a `nodeSelector` matches a
pool only if the class publishes the attributes and capacity it asks for.

## Co-scheduling and pools

A replica is a set of engines placed together on one cluster. Within a replica,
every member of a single engine is placed on **one** pool: each member carries
its own `nodeSelector`, but the scheduler requires a single pool that satisfies
them all.

It works this way because a gang's members coordinate over their pool's
interconnect fabric, and the scheduler can't reason about fabric. Pool identity
is the finest grain it has. An engine split across pools risks landing its
members on different fabrics. The collective then never forms, and the gang hangs
with no clear error. To avoid that, the scheduler never splits an engine: an engine that no
single pool satisfies isn't scheduled on that cluster. Different engines of the same replica
can use different pools, but all on the same cluster.

```mermaid
graph TD
    subgraph cluster ["One InferenceCluster"]
        subgraph pool1 ["Pool A"]
            L["prefill engine\nLeader + Worker\n(whole gang, one pool)"]
        end
        subgraph pool2 ["Pool B"]
            D["decode engine\nStandalone"]
        end
    end

    R["ModelReplica"] --> L
    R --> D
```

A member with no `nodeSelector` claims no devices. It matches the engine's pool
at no node cost and rides along on the gang's nodes, packed there by the
cluster's own scheduler.

## Counting capacity in nodes

Capacity is gated on **nodes**, not on individual GPUs. The only number the
scheduler reads from a member is its node cost:

```text
nodes = pods × copies
pods  = 1 for a Standalone or Leader, or worker.nodes for a Worker
```

A member that resolves no `claim: DRA` device, because it carried no
`nodeSelector` or matched only synthetic devices, costs zero nodes. The scheduler
sums the cost of a replica's members and places the replica only where every
engine's pool has enough free nodes, tracking a running ledger so it never
overcommits a cluster.

This accounting is deliberately coarse. The control-plane scheduler answers
"could this cluster plausibly host this replica," not "exactly which GPU does
each pod get." Device-level contention between deployments is left to DRA
admission on the workload cluster, which is authoritative: it rejects a pod whose
`ResourceClaim` can't be satisfied, and the next reconcile sees the updated
state.

## Pinning placement to a pool

The scheduler's pool choice is enforced, not advisory. Each scheduled pod carries
a Kubernetes `nodeSelector` on the `modelplane.ai/pool` node label, so it can only
land on the pool the scheduler chose. Without it, the cluster's scheduler could
place a pod on any pool whose devices match its DRA claim, and the fleet's
per-pool accounting would drift from where pods actually run.

Modelplane labels the nodes of every pool it provisions. On a BYO
(`source: Existing`) cluster it doesn't provision the nodes, so the operator must
label each pool's nodes `modelplane.ai/pool=<nodePools[].name>` themselves, or
worker pods for that pool stay `Pending`.

## Scaling, retention, and re-placement

Scheduling runs in two phases each reconcile:


- **Retain.** Each existing replica keeps its cluster if the cluster still exists
  and every member's pinned pool still matches its (possibly edited)
  `nodeSelector`. A degraded cluster, one that's not Ready or has no gateway
  address, is still retained; transient outages surface through the deployment's
  conditions, not re-placement.
- **Fill.** If the deployment wants more replicas than were retained, the
  shortfall is placed one at a time, each onto the eligible cluster hosting the
  fewest of this deployment's replicas, spreading before packing. If it wants
  fewer, the highest-index replicas are dropped first.


A replica never changes cluster. If its cluster is deleted, the replica stops
being emitted, Crossplane garbage-collects it, and the fill phase mints a fresh
replica elsewhere. Moving is always delete-plus-create, mirroring how Kubernetes
treats a pod whose node is gone.

## Known limitations

The scheduler is built to be conservative and predictable rather than optimal.
Two limits follow from that, both tracked for future work:

- **A whole node is charged per pod**
  ([#172](https://github.com/modelplaneai/modelplane/issues/172)). A pod that
  claims one GPU of an eight-GPU node still charges the whole node in the
  scheduler's accounting. This is safe, it can only under-count a pool's
  capacity, never overcommit it, but it can strand GPUs on deployments of
  sub-node engines.
- **An engine can't span pools, even on one fabric**
  ([#149](https://github.com/modelplaneai/modelplane/issues/149)). Because the
  scheduler has no concept of fabric, it refuses to split a gang across pools at
  all. That forecloses a legitimate case, GPU workers on one pool and a no-GPU
  coordinator on another within the same fabric, until fabric-aware placement
  lands.



---

# How Modelplane works

Source: /overview/how-it-works/


Modelplane runs as a control plane on its own cluster, the **control cluster**,
above the **inference clusters** that actually serve models. It's built on
[Crossplane](https://crossplane.io): platform teams and developers describe what
they want as Kubernetes resources, and Modelplane continuously reconciles the
fleet to match, composing the clusters, scheduling replicas, and exposing
endpoints. This page is the full tour. It covers the architecture and resources, then walks through what happens when you deploy a model.

## Modelplane API

Modelplane's API is two sets of resources, one per team, with everything in
between filled in for you. Platform teams describe the fleet, ML teams describe a
model, and Modelplane composes the rest.


  
    Platform team creates
    
      InferenceGateway
      The unified, OpenAI-compatible entry point on the control cluster.
    
    
      InferenceClass
      A tested hardware recipe for a node pool: the devices it offers and how to provision it.
    
    
      InferenceCluster
      A Kubernetes cluster in the fleet, provisioned by Modelplane or brought as-is.
    
  
  
    ML team creates
    
      ModelDeployment
      A model to serve, with engines, replica count, and an optional cache.
    
    
      ModelService
      One OpenAI-compatible endpoint, load-balanced across the endpoints it selects.
    
    
      ModelCache
      Model weights staged once per cluster on shared storage.
    
  
  
    Modelplane composes
    
      ModelReplica
      One complete serving instance on a specific cluster.
    
    
      ModelEndpoint
      A reachable endpoint, one per replica or set manually for an external provider.
    
  


The hierarchy mirrors Kubernetes core one scope up: `ModelDeployment` →
`ModelReplica` → `ModelService` → `ModelEndpoint` parallels `Deployment` → `Pod` → `Service` →
`Endpoint`, across a fleet instead of within a single cluster.

## What the control plane reconciles

Once the resources exist, Modelplane keeps the fleet matching them. Five concerns
run continuously:

1. **Provisioning.** From an `InferenceCluster`, Modelplane creates a full cluster 
   and its GPU node pools, or brings in a cluster you already run on
   any Kubernetes, and installs the serving stack on each.
2. **Scheduling.** A two-level scheduler places work: it pins each `ModelReplica`
   to a cluster and pool whose hardware meets the model's requirements, then the
   cluster's own scheduler binds the GPUs to the serving pods through DRA.
3. **Autoscaling.** Replicas are the scaling axis. Scaling a `ModelDeployment`'s
   `spec.replicas` adds or removes whole serving instances through the standard
   Kubernetes scale subresource, so `kubectl scale` or a KEDA `ScaledObject` work
   out of the box.
4. **Routing.** A `ModelService` exposes one OpenAI-compatible endpoint through
   the gateway and load-balances across the deployment's `ModelEndpoints`,
   wherever their replicas run. `ModelEndpoints` can also point at external
   inference services.
5. **Caching.** A `ModelCache` stages model weights on cluster storage once, so
   serving pods read them locally instead of re-downloading on every start.

## Universal compatibility

Modelplane is deliberately unopinionated about the engine. A `ModelDeployment`
describes the *shape* of a deployment, how many pods, on how many nodes, with
which devices, and nothing about how the engine runs internally. The engine flags
you write carry parallelism (tensor, pipeline, data, expert), quantization, and KV
transfer; Modelplane never injects them.

This is what lets one API serve any container-based engine and any topology
without special cases. Modelplane composes the engine onto the right cluster
resource and injects almost nothing, just the address a multi-node leader is
reachable at, so a worker can join it. New engines and new parallelism strategies
work without a change to Modelplane. The community publishes recipes (worked, copyable
manifests) to bridge the gap that flexibility leaves, rather than hard-coding
choices into the API.

## Fleet scheduler

For each replica, the scheduler picks a `(cluster, pool)` in two steps:

1. **Filter clusters** by `clusterSelector.matchLabels` against the standard
   Kubernetes labels on each `InferenceCluster`, the organizational metadata:
   tier, region, provider, compliance posture.
2. **Filter pools** by matching each device request in the deployment's
   `nodeSelector.devices` against the pool's `InferenceClass`. A request is based
   on DRA: a `count` and CEL selectors over a device's attributes and capacity, like
   "a GPU with at least 141Gi of memory." A pool fits when it has the devices the
   model asks for and enough free nodes to hold a replica.

Capacity is accounted at the node level across the fleet, so Modelplane never
overcommits a pool. Replicas are pinned to their cluster once placed and stay
there across reconciles; if a cluster is deleted, the scheduler re-places its
replicas elsewhere. [How it schedules](/architecture/scheduling/)
covers the placement rules and their limits in full.

## Deploying a model

Creating a `ModelDeployment` kicks off the loop end to end. The scheduler
discovers the ready clusters (filtered by your label selector if you set one),
matches each engine's device requests against their pools, and pins each replica
to a cluster that fits. Modelplane composes a `ModelReplica` on each chosen
cluster, turns it into the right serving workload there, creates a `ModelEndpoint`
per replica, and your `ModelService` routes traffic across them through one stable
endpoint on the gateway. Scale the deployment up or down and the same loop
re-converges.

## Serving topologies

A single-node deployment composes to a Kubernetes Deployment fronted by a
service. When a model is too large for one node, an engine becomes a gang: a
`Leader` member and one or more `Worker` members that Modelplane composes into a
LeaderWorkerSet, serving the model together across nodes. Gang deployments
should stage their weights through a `ModelCache`, so the pods share one copy
instead of each pulling the same model.

Disaggregated serving splits prefill and decode into separate engines
(`serving.mode: PrefillDecode`) that run on the same cluster and hand off the KV
cache between them. Modelplane wires up the cluster-edge routing that pairs each
request's prefill and decode; the engines carry the KV-transfer flags. Both are
described in full in the [model deployment docs](/models/model-deployment/).

## Next steps




  
    
    FAQ
  
  Quick answers on how Modelplane compares and what it requires.
  →



  
    
    Get started
  
  Put it together: deploy Modelplane and serve a model.
  →








---

# Qwen3-Coder-480B

Source: /recipes/qwen3-coder/


A 480B code MoE (35B active). Two validated shapes: the BF16 weights span two
H200 nodes as a gang over EFA, served from a `ModelCache`; the FP8 checkpoint
fits one node, so it runs as a single `Standalone` engine on SGLang with no
cache.

Both shapes were run end to end; the `InferenceClass` and `ModelDeployment` are
the exact manifests from those runs. Apply the platform side first, then the ML
side. The `InferenceCluster` carries an EC2 capacity reservation placeholder to
edit before applying.

## Validated deployments









  MoE
  480B A35B
  32,768 ctx
  vLLMSGLang


  
    Cloud
    
      
        
          
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      8× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        BF16
      
        FP8
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.23.0lmsysorg/sglang:v0.5.10.post1-runtime
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/qwen3-coder
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform










  
  
  
  
    Multi-node (BF16)
    
    
  
  
  
  
  
    Single-node (FP8)
    
    
  
  


  








  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the H200 shape, validated serving Qwen3-Coder-480B
# multi-node on EKS. 8x NVIDIA H200 on an EKS p5en.48xlarge, with EFA.
#
# Both the GPU and the EFA fabric are claim: DRA devices. A multi-node gang's
# nodeSelector requests both, so the scheduler co-schedules the whole gang on a
# pool that has them and DRA binds 8 GPUs + 16 EFA interfaces per pod. The EFA
# device is installed by the EFA DRA driver (DRANET) in the serving stack.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-h200-8x-p5en
spec:
  description: "EKS p5en.48xlarge, 8x NVIDIA H200, EFA"
  provisioning:
    provider: EKS
    eks:
      instanceType: p5en.48xlarge
      diskSizeGb: 1024
      accelerator:
        type: nvidia-h200
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
    capacity:
      memory: { value: "140Gi" }   # advertised below the ~141 GiB the driver reports
  - name: efa
    claim: DRA
    driver: dra.net
    deviceClassName: efa.networking.k8s.aws
    count: 16

  


  
    
      inference-cluster.yaml
      
          
          
        
      
    
    # An EKS InferenceCluster with a two-node H200 pool over EFA, validated serving
# Qwen3-Coder-480B as a multi-node gang. The H200 nodes come from an EC2
# Capacity Block reserved for ML.
#
# fabric.type: EFA turns on Elastic Fabric Adapter for the gang's cross-node
# traffic; without it multi-node NCCL falls back to TCP, which is slow and
# unstable.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-coder
  labels:
    modelplane.ai/region: us
spec:
  cluster:
    source: EKS
    eks:
      region: us-east-2
  nodePools:
  - name: gpu-h200
    className: eks-h200-8x-p5en
    nodeCount: 2
    minNodeCount: 2
    maxNodeCount: 2
    zones:
    - us-east-2b
    fabric:
      type: EFA
    capacityBlock:
      capacityReservationId: cr-0123456789abcdef0  # replace with your reservation ID

  


  
    bash
    
      
        
        
      
    
  
  curl -fsSL /examples/recipes/qwen3-coder/inference-cluster.yaml \
  | sed 's/cr-0123456789abcdef0//' \
  | kubectl apply -f -











  
    
      inference-class-fp8.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the H200 shape without EFA, validated serving the FP8
# Qwen3-Coder-480B checkpoint single-node on SGLang.
#
# The FP8 weights (~480 GB) fit on one 8x H200 node, so this needs no second
# node, no fabric, and no ModelCache - the GPU is the only device.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-h200-8x-p5en
spec:
  description: "EKS p5en.48xlarge, 8x NVIDIA H200"
  provisioning:
    provider: EKS
    eks:
      instanceType: p5en.48xlarge
      diskSizeGb: 1024
      accelerator:
        type: nvidia-h200
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
    capacity:
      memory: { value: "140Gi" }   # advertised below the ~141 GiB the driver reports

  







## Deployment










  
  
  
  
    Multi-node (BF16)
    
    
  
  
  
  
  
    Single-node (FP8)
    
    
  
  


  








  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The shared, read-write-many cache the multi-node gang serves from. Hydrated
# once per matched cluster from the gated Hugging Face repo; every gang pod
# mounts it at /mnt/models. ~960 GB of BF16 weights, so sizeGiB leaves headroom.
#
# The repo is gated, so it needs a Hugging Face token. Create the authSecret once
# in the ModelCache's namespace on the control plane; Modelplane propagates it to
# each matched cluster.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    authSecret:
      name: hf-token
      key: HF_TOKEN
    sizeGiB: 1100

  


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen3-Coder-480B served BF16 across two H200 nodes, validated end to end on
# EKS over EFA. A 480B MoE doesn't fit one node, so the engine is a Leader +
# Worker gang spanning two nodes via LeaderWorkerSet, both pods mounting the
# shared ModelCache at /mnt/models.
#
# Each member requests 8 GPUs + 16 EFA interfaces per node; the scheduler
# co-schedules the gang on the H200 pool. A member joins the leader through
# $(MODELPLANE_LEADER_ADDRESS) and takes its rank from $(MODELPLANE_RANK),
# both injected by Modelplane.
#
# Notes on the engine flags:
#   --distributed-executor-backend=mp with --nnodes/--node-rank/--master-addr/
#     --headless is vLLM's native multiprocessing multi-node path.
#     vllm/vllm-openai:v0.23.0 no longer ships Ray, so the Ray-based
#     multi-node-serving.sh helper doesn't work on this image; the MP backend
#     needs nothing extra.
#   TP8 x PP2: tensor-parallel within a node over NVLink, pipeline-parallel
#     across the two nodes. tensor-parallel-size = GPUs per node,
#     pipeline-parallel-size = nodes.
#   --tool-call-parser=qwen3_xml is the parser for Qwen3-Coder specifically
#     (the dense Qwen3 models use hermes). The model is non-thinking, so there's
#     no reasoning parser.
#   --max-model-len=32768 caps context to fit; the native 256K isn't needed.
#   FI_PROVIDER=efa / NCCL_DEBUG=INFO point NCCL at the EFA fabric.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us
      modelCacheRef:
        name: qwen3-coder
      engines:
      - name: qwen3-coder
        members:
        - role: Leader
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
            - name: efa
              count: 16
              selectors:
              - cel: |
                  device.driver == "dra.net"
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                env:
                - name: NCCL_DEBUG
                  value: "INFO"
                - name: FI_PROVIDER
                  value: "efa"
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve /mnt/models
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=$(MODELPLANE_RANK)
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --max-model-len=32768
                  --gpu-memory-utilization=0.92
                  --enable-auto-tool-choice
                  --tool-call-parser=qwen3_xml
                  --port=8000
        - role: Worker
          worker:
            nodes: 1
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
            - name: efa
              count: 16
              selectors:
              - cel: |
                  device.driver == "dra.net"
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                env:
                - name: NCCL_DEBUG
                  value: "INFO"
                - name: FI_PROVIDER
                  value: "efa"
                command:
                - /bin/sh
                - -c
                - >-
                  exec vllm serve /mnt/models
                  --served-model-name=qwen3-coder
                  --tensor-parallel-size=8
                  --pipeline-parallel-size=2
                  --distributed-executor-backend=mp
                  --nnodes=2 --node-rank=$(MODELPLANE_RANK)
                  --master-addr=$(MODELPLANE_LEADER_ADDRESS)
                  --headless
                  --max-model-len=32768
                  --gpu-memory-utilization=0.92

  


  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the multi-node BF16 qwen3-coder deployment as a single
# OpenAI-compatible URL. Read the public address from status.address:
#   kubectl get ms qwen3-coder -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-coder

  











  
    
      model-deployment-fp8.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen3-Coder-480B served FP8 on a single 8x H200 node with SGLang, validated
# end to end on EKS. The FP8 checkpoint (~480 GB) fits one node, so this is a
# single Standalone engine: no second node, no EFA, no ModelCache. The engine
# pulls the public FP8 repo straight to the node's local disk.
#
# SGLang-specific notes:
#   --ep-size 8 is required, not optional. Pure --tp-size 8 fails at FP8 weight
#     creation ("output_size ... not divisible by ... block_n = 128"): the
#     block-FP8 MoE doesn't shard evenly across 8 tensor-parallel ranks. Expert
#     parallelism shards whole experts and gets past it.
#   --tool-call-parser qwen3_coder is SGLang's parser name for this model
#     (vLLM's is qwen3_xml). The model is non-thinking, so no reasoning parser.
#   Image tag matters: lmsysorg/sglang v0.5.11-v0.5.13(.post1) -runtime images
#     are broken (ModuleNotFoundError: distro). v0.5.10.post1-runtime is the
#     most recent clean tag with Qwen3-Coder support.
#   --host 0.0.0.0 --port 8000: SGLang defaults to 127.0.0.1:30000, but
#     Modelplane's contract is 0.0.0.0:8000 with a /health probe. Args pass
#     through verbatim - Modelplane injects nothing for a non-vLLM engine.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen3-coder-sgl
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us
      engines:
      - name: qwen3-coder-sgl
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("120Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: lmsysorg/sglang:v0.5.10.post1-runtime
                command:
                - /bin/sh
                - -c
                - >-
                  exec python3 -m sglang.launch_server
                  --model-path Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8
                  --served-model-name qwen3-coder
                  --tp-size 8
                  --ep-size 8
                  --context-length 32768
                  --page-size 32
                  --trust-remote-code
                  --tool-call-parser qwen3_coder
                  --host 0.0.0.0
                  --port 8000

  


  
    
      model-service-fp8.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the single-node FP8 qwen3-coder-sgl deployment as a single
# OpenAI-compatible URL. Read the public address from status.address:
#   kubectl get ms qwen3-coder-sgl -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen3-coder-sgl
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen3-coder-sgl

  









---

# GLM-4.5-Air

Source: /recipes/glm-4.5-air/


A 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of
vLLM, on a single A100 40 GB. Modelplane treats the engine as any
OpenAI-compatible container, so the only changes from a vLLM deployment are
the image and args: the container is still named `engine` and listens on
`:8000`. vLLM can't load this Unsloth quantization format. llama.cpp can, and
`-hf` pulls the checkpoint straight from Hugging Face at startup, so a
one-time deployment needs no `ModelCache`.

The model is bigger than one A100's VRAM, so `--n-cpu-moe` offloads the MoE
expert tensors to host RAM and the GPU runs the active path and KV cache.
That's how a 106B model fits one A100 instead of a multi-GPU node. Apply the
platform side first, then the ML side.

## Validated deployments









  MoE
  106B A12B
  8,192 ctx
  llama.cpp


  
    Cloud
    
      
        
          


AWS
        
      
        
          
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        GGUF IQ4_XS
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      ghcr.io/ggml-org/llama.cpp:server-cuda
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/glm-4.5-air
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A single A100 40GB on GKE. GLM-4.5-Air is a ~106B MoE; a 4-bit GGUF doesn't
# fit one A100's VRAM, but llama.cpp offloads the expert tensors to host RAM
# (see the ModelDeployment's --n-cpu-moe), so the GPU only holds the active path
# + KV cache. One A100, not a multi-GPU node.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
    capacity:
      memory: { value: "40960Mi" }

  




  
    
      inference-cluster.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A GKE cluster with a single A100 node offering the class above.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-a100
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west1-b

  



## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # GLM-4.5-Air (~106B MoE) served from an Unsloth GGUF via llama.cpp instead of
# vLLM, on a SINGLE A100. Modelplane treats the engine as any OpenAI-compatible
# container, so the only changes from a vLLM deployment are the image and args:
# the container is still named `engine` and listens on :8000. vLLM can't load
# Unsloth's UD- dynamic quants; llama.cpp can, and `-hf` pulls the quant from
# HuggingFace at startup (no ModelCache needed for a one-off).
#
# The model is bigger than one A100's VRAM, so --n-cpu-moe offloads the MoE
# expert tensors to host RAM; the GPU runs the active path. That's how a 106B
# model fits one A100 instead of a multi-GPU node. --port 8000 because llama.cpp
# defaults to 8080 and Modelplane scrapes 8000.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: glm-air
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: glm
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: ghcr.io/ggml-org/llama.cpp:server-cuda
                args:
                - "-hf"
                - "unsloth/GLM-4.5-Air-GGUF:IQ4_XS"
                - "--host"
                - "0.0.0.0"
                - "--port"
                - "8000"
                - "-ngl"
                - "999"
                - "--n-cpu-moe"
                - "99"
                - "--jinja"
                - "-c"
                - "8192"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms glm-air -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: glm-air
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: glm-air

  





---

# Cache Model Weights

Source: /models/model-cache/


**API:** [`modelplane.ai/v1alpha1` · ModelCache](/reference/modelcaches/)

A `ModelCache` stages a model's weights on shared workload-cluster storage,
fetched once from the configured source rather than downloaded again on every
pod start. `ModelDeployments` reference a cache via
`spec.template.spec.modelCacheRef.name`, and Modelplane mounts it at
`/mnt/models` in every serving pod, shared across the
pods of a multi-node engine. The engine reads weights locally from the mount.

`ModelCache` is recommended for multi-node deployments and optional for
single-node cold-start optimization.

## What to cache

The required `source` enum names the kind, with the matching source object set
alongside it. Setting `source: HuggingFace` selects `spec.huggingFace`, which
carries the `repo` to fetch, an optional `revision` (branch, tag, or commit), and
`sizeGiB`, how much storage the weights get on each cluster. Size it to the
model, since a value below the model's size leaves no room to stage the weights.
`HuggingFace` is the only source today.

The cache mounts at `/mnt/models` on every consuming pod, so the engine's args
reference that path (`--model=/mnt/models` for vLLM) rather than the source.

## Authenticating

A gated or private model needs a credential to fetch. When a cache stages the
weights, the credential lives on the cache: set `authSecret` to name a Secret in
the cache's namespace, and Modelplane propagates it to every cluster the cache
stages to, for the hydration to read.

Create the Secret once on the control plane, then reference it:

```bash
kubectl create secret generic hf-token \
  --namespace ml-team \
  --from-literal=HF_TOKEN=hf_xxxxxxxx
```

```yaml {nocopy=true}
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    authSecret:
      name: hf-token         # a Secret in this ModelCache's namespace
      key: HF_TOKEN          # defaults to HF_TOKEN
    sizeGiB: 1100
```

Without a cache, the engine fetches the model itself at startup, so the
credential goes on the `ModelDeployment` instead, as `HF_TOKEN` in the engine
container's `env`.

## Where to cache

An optional `clusterSelector` scopes where the cache is staged. Omitting it
stages the cache on every cluster in the fleet; setting `matchLabels` restricts
it to clusters carrying those labels. A `ModelDeployment` that references the cache
places *new* replicas only onto clusters within this footprint, so narrowing the
selector also narrows where replicas can land: a replica never schedules to a
cluster the cache didn't stage to. Replicas already running are left where they
are.

## Loading from cache

A cache only pays off if the engine reads from it quickly. With its default
loader an engine can read a large model from shared storage slowly enough that
the cache makes cold starts *worse* than fetching the model directly, since you
pay to hydrate the cache and then wait on a slow read. Choose a fast loader with
your engine flags.

For vLLM on EKS, `--load-format=runai_streamer` reads from the EFS-backed cache
dramatically faster than the default loader (minutes rather than tens of
minutes for a large model), tuned further with `--model-loader-extra-config`:

```yaml {nocopy=true}
args:
- --model=/mnt/models
- --load-format=runai_streamer
- --model-loader-extra-config={"concurrency":16,"distributed":true}
```

The right loader and settings depend on the engine and the storage backend, so
treat these as a starting point and measure your own cold-start time. The
[Kimi-K2 recipe](/recipes/kimi-k2/) uses this configuration end to
end.

## Storage prerequisites


The cache PVC needs a `ReadWriteMany` (RWX) StorageClass on the workload cluster.
What the platform admin must set up depends on the cloud:


- **GKE** and **EKS:** auto-provisioned. Nothing for the admin to do.
- **Existing:** the admin sets up a `ReadWriteMany` StorageClass on the cluster.

Either way, your `ModelCache` and `ModelDeployment` specs are the same. How
storage is provided on each cluster source, and how to bring your own backend, is
covered in [Register a Cluster](/platform/inference-cluster/#cache-storage).

## Example


  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelCache stages a model artifact on workload-cluster storage as a
# first-class resource. Modelplane composes a ReadWriteMany PVC on each matched
# cluster and hydrates it once from the configured source. A ModelDeployment
# references it via spec.modelCacheRef; the PVC mounts at /mnt/models read-write
# into every serving pod, so the engine reads weights locally instead of
# fetching them at boot.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen3-coder
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen3-Coder-480B-A35B-Instruct
    # Gated repo, so a Hugging Face token is needed. Create this Secret once in
    # the ModelCache's namespace on the control plane; Modelplane propagates it
    # to each matched cluster.
    authSecret:
      name: hf-token
      key: HF_TOKEN
    sizeGiB: 1100
  # Optional: stage only on clusters matching these labels. Omit to stage on
  # every matched cluster. Narrowing this also narrows where a referencing
  # ModelDeployment can place new replicas.
  # clusterSelector:
  #   matchLabels:
  #     modelplane.ai/tier: frontier

  





---

# Deploying a model

Source: /getting-started/deploying-a-model/


Now that the platform is provisioned, the ML team can declare what a model needs
with a `ModelDeployment`. Describe the hardware requirements and the scheduler
schedules against the capacity the platform team published.

## Create a deployment

Create a namespace for the model:

```bash
kubectl create namespace ml-team
```

The device selector matches against the capacity declared in the
`InferenceClass`, not the pod's resource requests. Any L4 node satisfies
`>= 20Gi`, so this deployment runs on the cluster you just added:










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  


  








  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # Any L4 satisfies >= 20Gi. The selector matches against the capacity
          # declared in the InferenceClass, not the pod's resource requests.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # Any L4 satisfies >= 20Gi. The selector matches against the capacity
          # declared in the InferenceClass, not the pod's resource requests.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # Any L4 satisfies >= 20Gi. The selector matches against the capacity
          # declared in the InferenceClass, not the pod's resource requests.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-demo
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # Any L4 satisfies >= 20Gi. The selector matches against the capacity
          # declared in the InferenceClass, not the pod's resource requests.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  







Wait until `REPLICAS` shows `1`:

```bash
kubectl get md -n ml-team --watch
```

To see which cluster the scheduler chose:

```bash
kubectl get modelreplica -n ml-team
```

```shell{nocopy=true}
NAME              CLUSTER       SYNCED   READY   COMPOSITION                   AGE
qwen-demo-7323a   eks-us-east   True     True    modelreplicas.modelplane.ai   12m
```

The ML team never named a cluster. The scheduler matched the GPU requirement
(`>= 20Gi`) against the `InferenceClass` the platform team published and made
the placement. 

## Expose the model

A `ModelService` selects `ModelEndpoints` by label and creates a Gateway API
`HTTPRoute` that routes to them. Modelplane creates one `ModelEndpoint` per
replica, labeled with the deployment name:


  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelService exposes one or more ModelDeployments via a single
# OpenAI-compatible endpoint. It composes a Gateway-API HTTPRoute on the
# control plane that load-balances across every ModelEndpoint matching
# its selector.
#
# Modelplane composes one ModelEndpoint per ModelReplica, labeled
# `modelplane.ai/deployment: <deployment-name>`. So a ModelService with
# that label selector reaches every replica of the named deployment.
#
# Once the service is ready, its public address is on status.address:
#   kubectl get ms qwen -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-demo

  



The request path is `/<namespace>/<modelservice-name>/...` (`/ml-team/qwen/` in
this example), from the `ModelService` named `qwen`. The `model` field in the
request body is the Hugging Face id `Qwen/Qwen2.5-0.5B-Instruct`, since this
deployment doesn't set `--served-model-name`.

## Send a request

Read the endpoint's public address from the `ModelService` status:

```bash
ADDRESS=$(kubectl get ms qwen -n ml-team -o jsonpath='{.status.address}')
```

Send a request to it:

```bash
kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -v "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'
```

The request routes to the replica on the cluster Modelplane placed it on.
You should get a response in a few seconds:

```json {nocopy=true}
{
  "id": "chatcmpl-c88b1429-067d-40a5-971c-ab9c54153c26",
  "model": "Qwen/Qwen2.5-0.5B-Instruct",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "Kubernetes (K8s) is an open-source platform for automating 
        the deployment, scaling, and management of containerized applications. 
        It provides scalable orchestration capabilities that enable developers 
        to deploy complex applications quickly and efficiently across various environments."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 37,
    "completion_tokens": 48,
    "total_tokens": 85
  }
}

```

## Next step

The platform team declared capacity and in this guide the ML team deployed a
model behind a stable endpoint. Neither team needed to know what the other was doing. Modelplane matched them.

In the next step, the platform team grows the fleet. [Scale the platform](/getting-started/scale-the-platform/) to add more clusters across regions.



---

# Kimi-K2

Source: /recipes/kimi-k2/


A 1T MoE (1 trillion parameters) served prefill/decode disaggregated across two
H200 nodes: two engines, one per phase, with Modelplane composing the llm-d
routing layer between them. This recipe serves an INT4 quantization of the
model; the native FP8 weights need four such nodes.

This recipe was run end to end; the `InferenceClass` and `ModelDeployment` are
the exact manifests from that run. Apply the platform side first, then the ML
side. The `InferenceCluster` carries an EC2 capacity reservation placeholder to
edit before applying.

## Validated deployments









  MoE
  1T A32B
  131,072 ctx
  vLLM


  
    Cloud
    
      
        
          
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      8× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        INT4
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.23.0
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/kimi-k2
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the H200 shape, validated serving Kimi-K2 prefill/decode
# disaggregated on EKS. 8x NVIDIA H200 on an EKS p5en.48xlarge, with EFA.
#
# Both the GPU and the EFA fabric are claim: DRA devices. The two P/D engines
# each request 8 GPUs + 16 EFA interfaces, and the scheduler places one on each
# H200 node; NIXL ships KV cache between them over EFA.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-h200-8x-p5en
spec:
  description: "EKS p5en.48xlarge, 8x NVIDIA H200, EFA"
  provisioning:
    provider: EKS
    eks:
      instanceType: p5en.48xlarge
      diskSizeGb: 1024
      accelerator:
        type: nvidia-h200
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
    capacity:
      memory: { value: "140Gi" }   # advertised below the ~141 GiB the driver reports
  - name: efa
    claim: DRA
    driver: dra.net
    deviceClassName: efa.networking.k8s.aws
    count: 16

  




  
    
      inference-cluster.yaml
      
          
          
        
      
    
    # An EKS InferenceCluster with a two-node H200 pool over EFA, validated serving
# Kimi-K2 as a prefill/decode pair (one engine per node). The H200 nodes come
# from an EC2 Capacity Block reserved for ML.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-kimi
  labels:
    modelplane.ai/region: us
spec:
  cluster:
    source: EKS
    eks:
      region: us-east-2
  nodePools:
  - name: gpu-h200
    className: eks-h200-8x-p5en
    nodeCount: 2
    minNodeCount: 2
    maxNodeCount: 2
    zones:
    - us-east-2b
    fabric:
      type: EFA
    capacityBlock:
      capacityReservationId: cr-0123456789abcdef0  # replace with your reservation ID

  




  
    bash
    
      
        
        
      
    
  
  curl -fsSL /examples/recipes/kimi-k2/inference-cluster.yaml \
  | sed 's/cr-0123456789abcdef0//' \
  | kubectl apply -f -



## Deployment


  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The shared, read-write-many cache both P/D engines serve from. Hydrated once
# per matched cluster; both phases mount the same RWX volume at /mnt/models.
#
# This validated run served an INT4 quantization of Kimi K2 rather than the
# native 1T-parameter FP8 model, which would need four 8x H200 nodes. The quant
# repo is public, so no authSecret is needed here.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: RedHatAI/Kimi-K2-Instruct-quantized.w4a16
    sizeGiB: 600

  




  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Kimi-K2 served prefill/decode disaggregated across two H200 nodes, validated
# end to end on EKS. serving.mode: PrefillDecode makes the two engines below one
# P/D pair: Modelplane composes the llm-d routing layer (InferencePool, endpoint
# picker, the NIXL pd-sidecar) between them. Each engine is a single-node 8-GPU
# Standalone pod; the scheduler places prefill on one node and decode on the
# other, and KV cache ships between them over EFA via NIXL.
#
# Notes on the engine flags (most P/D machinery is Modelplane's; the engine
# config has real sharp edges):
#   EP, not TP8. --tensor-parallel-size=1 --data-parallel-size=8
#     --enable-expert-parallel is vLLM's DeepSeek-V3 / Kimi single-node recipe
#     (Kimi-K2 is DeepSeek-V3 arch). It keeps each expert whole on a GPU and
#     dodges the Marlin %128 alignment trap that a plain TP8 layout would hit.
#   --load-format=runai_streamer cold-reads ~509 GiB per engine off the shared
#     RWX cache in ~6 minutes (vs ~45 with the default loader).
#   --tokenizer / --override-generation-config are workarounds for two bugs in
#     this specific quant repo (an off-by-2 in its bundled tokenizer, and a
#     wrong eos_token_id), not normal flags. The override pulls a gated repo at
#     startup, so HF_TOKEN is set on both engines.
#   The decode engine runs on :8001; the pd-sidecar owns :8000.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: kimi-k2
      serving:
        mode: PrefillDecode        # the two engines below are one P/D pair
      engines:
      - name: kimi-prefill
        phase: Prefill
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("130Gi")) >= 0
            - name: efa
              count: 16
              selectors:
              - cel: |
                  device.driver == "dra.net"
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command: ["vllm", "serve", "/mnt/models"]
                args:
                - --served-model-name=kimi-k2
                - --quantization=compressed-tensors
                - --tensor-parallel-size=1
                - --data-parallel-size=8
                - --enable-expert-parallel
                - --block-size=64
                - --max-model-len=131072
                - --trust-remote-code
                - --tool-call-parser=kimi_k2
                - --enable-auto-tool-choice
                - --load-format=runai_streamer
                - --model-loader-extra-config={"concurrency":16,"distributed":true}
                - --tokenizer=moonshotai/Kimi-K2-Instruct
                - --override-generation-config={"eos_token_id":163586}
                - --port=8000
                - --kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_producer"}
                env:
                - name: HF_TOKEN
                  valueFrom:
                    secretKeyRef: {name: hf-token, key: HF_TOKEN}
      - name: kimi-decode
        phase: Decode
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("130Gi")) >= 0
            - name: efa
              count: 16
              selectors:
              - cel: |
                  device.driver == "dra.net"
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                command: ["vllm", "serve", "/mnt/models"]
                args:
                - --served-model-name=kimi-k2
                - --quantization=compressed-tensors
                - --tensor-parallel-size=1
                - --data-parallel-size=8
                - --enable-expert-parallel
                - --block-size=64
                - --max-model-len=131072
                - --trust-remote-code
                - --tool-call-parser=kimi_k2
                - --enable-auto-tool-choice
                - --load-format=runai_streamer
                - --model-loader-extra-config={"concurrency":16,"distributed":true}
                - --tokenizer=moonshotai/Kimi-K2-Instruct
                - --override-generation-config={"eos_token_id":163586}
                - --port=8001                                  # pd-sidecar owns 8000
                - --kv-transfer-config={"kv_connector":"NixlConnector","kv_role":"kv_consumer"}
                env:
                - name: HF_TOKEN
                  valueFrom:
                    secretKeyRef: {name: hf-token, key: HF_TOKEN}

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the kimi-k2 deployment as a single OpenAI-compatible URL. Read the
# public address from status.address:
#   kubectl get ms kimi-k2 -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: kimi-k2

  





---

# Register a Cluster

Source: /platform/inference-cluster/

**API:** [`modelplane.ai/v1alpha1` · InferenceCluster](/reference/inferenceclusters/)

An `InferenceCluster` represents a Kubernetes cluster configured for model
serving. Platform teams create these to provide GPU capacity.


Each cluster has:

- A **cluster source**: `GKE`, `EKS`, `AKS`, `Nebius` or `Vultr` (Modelplane provisions
  the full cluster) or `Existing` (bring a cluster you manage yourself). See
  [Supported Providers](/platform/providers/) for the clouds and
  neoclouds Modelplane runs on.
- One or more **node pools**, each referencing an `InferenceClass` for its
  hardware capabilities and provisioning recipe.
- **Labels** for organizational metadata: tier, region, provider. These are the
  matching surface for `ModelDeployment.clusterSelector`.

Modelplane installs the serving stack it needs on every cluster it manages,
including existing clusters, which it assumes are solely for its use.

## Ownership and requirements

Modelplane assumes exclusive ownership of every `InferenceCluster`. The fleet
scheduler's capacity accounting relies on Modelplane being the only thing placing
GPU workloads on the cluster, so dedicate each cluster to Modelplane rather than
sharing it with other workloads.

Modelplane also has opinions about how a cluster is set up: its Kubernetes
version, the components it installs, and required features like DRA for binding
GPUs to pods. On provisioned clusters Modelplane handles this for you. On an
existing cluster the platform team must meet the requirements.

## Provisioned and existing clusters

The `cluster.source` discriminator picks one of two models:

- **Provisioned (`GKE`, `EKS`, `AKS`, `Nebius`, `Vultr`).** Modelplane creates the cluster and its GPU node
  pools from each pool's `InferenceClass`, labels the pool's nodes so the
  scheduler's placement is enforced, and provisions the storage class for model
  weights. It also injects a non-GPU **system pool** with opinionated defaults to
  run the inference stack, so you only declare the GPU pools you want.
- **Existing (`Existing`).** A kubeconfig `Secret` provides access to a cluster
  you run yourself. Modelplane installs the serving stack it needs but doesn't
  provision infrastructure, and each pool's `InferenceClass` provides hardware
  capabilities for scheduling only. You're responsible for the cluster meeting
  Modelplane's requirements, including labeling each pool's nodes
  `modelplane.ai/pool=<pool-name>` (see
  [how scheduling pins placement](/architecture/scheduling/#pinning-placement-to-a-pool)).

## Examples










  
  
  
  
    GKE
    
    
  
  
  
  
  
    EKS
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  
  
  
  
    Vultr
    
    
  
  
  
  
  
    Existing
    
    
  
  


  








  
    
      inference-cluster-gke.yaml
      
          
          
        
      
    
    # An InferenceCluster backed by a GKE cluster.
#
# Modelplane provisions the full GKE cluster (VPC, subnet, system pool,
# GPU pools, service account, IAM bindings) and installs the inference
# stack (cert-manager, Envoy Gateway, Prometheus, LeaderWorkerSet,
# Gateway API).
#
# The system pool that hosts control-plane components is provisioned
# automatically and is not declared here. Only GPU pools - each
# referencing an InferenceClass that describes the hardware shape and
# how to provision it - need to be declared.
#
# Add labels to this InferenceCluster to control which deployments land on it
# via a ModelDeployment's clusterSelector.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-us-central
  labels:
    modelplane.ai/region: us-central
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1

  nodePools:
  - name: gpu-l4
    className: gke-l4-1x-g2
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 4
    zones:
    - us-central1-a
    - us-central1-c

  











  
    
      inference-cluster-eks.yaml
      
          
          
        
      
    
    # An InferenceCluster backed by an EKS cluster.
#
# Modelplane provisions the full EKS cluster (VPC, subnets, internet
# gateway, IAM roles for the cluster and nodes, system + GPU node
# groups, vpc-cni / kube-proxy / coredns addons) and installs the
# inference stack (cert-manager, Envoy Gateway, Prometheus,
# LeaderWorkerSet, Gateway API).
#
# The system node group that hosts control-plane components is
# provisioned automatically and is not declared here. Only GPU node
# groups - each referencing an InferenceClass that describes the
# hardware shape and how to provision it - need to be declared.
#
# Modelplane provisions EFS RWX storage for ModelCache on EKS: an
# Elastic-throughput file system, mount targets, the EFS CSI driver, and
# a 'modelplane-rwx-efs' StorageClass pinned to it. The admin does
# nothing, and provisioned EKS clusters take no StorageClass override.
#
# Delete this InferenceCluster with foreground cascading deletion for a
# clean teardown:
#
#   kubectl delete inferencecluster eks-us-west --cascade=foreground
#
# The inference stack runs on the EKS cluster and must uninstall while
# the cluster's API server and kubeconfig still exist - otherwise its
# Helm releases hang, and a load balancer one of them created can leak
# its security group and block the VPC from deleting. Foreground
# deletion holds the cluster until the stack is uninstalled. Background
# deletion (the kubectl default) tears everything down at once and can
# orphan cloud resources.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2

  nodePools:
  - name: gpu-l4
    className: eks-l4-1x-g6
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 4
    zones:
    - us-west-2a
    - us-west-2b

  











  
    
      inference-cluster-aks.yaml
      
          
          
        
      
    
    # An InferenceCluster backed by an Azure AKS cluster.
#
# Modelplane provisions the full AKS cluster (a dedicated resource group,
# virtual network and subnet, control plane with a system-assigned managed
# identity, system + GPU node pools) and installs the inference stack
# (cert-manager, Envoy Gateway, Prometheus, LeaderWorkerSet, Gateway API).
#
# The system node pool that hosts control-plane components is
# provisioned automatically and is not declared here. Only GPU node
# pools - each referencing an InferenceClass that describes the
# hardware shape and how to provision it - need to be declared. AKS
# caps pool names at 12 lowercase alphanumeric characters.
#
# The cluster's kubeconfig embeds a client certificate for a
# cluster-admin local account, so Modelplane authenticates to the
# cluster with the kubeconfig alone - no separate cloud identity, like
# EKS and unlike Nebius.
#
# ModelCache RWX storage is served by Azure Files: AKS bundles the
# Azure Files CSI driver, so Modelplane only creates the
# modelplane-rwx-fs StorageClass (Premium)
# and the driver provisions the backing storage account on demand.
#
# Delete this InferenceCluster with foreground cascading deletion for a
# clean teardown:
#
#   kubectl delete inferencecluster aks-westeurope --cascade=foreground
#
# The inference stack runs on the AKS cluster and must uninstall while
# the cluster's API server and kubeconfig still exist - otherwise its
# Helm releases hang, and a load balancer one of them created can leak
# and block the network from deleting. Foreground deletion holds the
# cluster until the stack is uninstalled. Background deletion (the
# kubectl default) tears everything down at once and can orphan cloud
# resources.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-westeurope
  labels:
    modelplane.ai/region: westeurope
spec:
  cluster:
    source: AKS
    aks:
      location: westeurope

  nodePools:
  - name: gpuh100
    className: aks-h100-8x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 4

  











  
    
      inference-cluster-nebius.yaml
      
          
          
        
      
    
    # An InferenceCluster backed by a Nebius mk8s cluster.
#
# Modelplane provisions the full mk8s cluster (VPC network and subnet,
# control plane with a public endpoint, system + GPU node groups, GPU
# clusters for InfiniBand fabrics) and installs the inference stack
# (cert-manager, Envoy Gateway, Prometheus, LeaderWorkerSet, Gateway
# API).
#
# The system node group that hosts control-plane components is
# provisioned automatically and is not declared here. Only GPU node
# pools - each referencing an InferenceClass that describes the
# hardware shape and how to provision it - need to be declared.
#
# Nebius authenticates every Kubernetes client through Nebius IAM: the
# mk8s kubeconfig carries only the cluster endpoint and CA certificate,
# and Nebius has no API that mints service account keys server-side. So
# Modelplane reuses the credentials Secret of the Nebius
# ClusterProviderConfig named default - the same identity that
# provisions the cluster - to authenticate to it. Nothing to create
# beyond the ClusterProviderConfig the provider needs anyway.
#
# ModelCache RWX storage is provisioned automatically: a Nebius shared
# filesystem mounted on every node group, served through Nebius's
# csi-mounted-fs-path CSI driver as the modelplane-rwx-fs StorageClass.
#
# Delete this InferenceCluster with foreground cascading deletion for a
# clean teardown:
#
#   kubectl delete inferencecluster nebius-eu-north --cascade=foreground
#
# The inference stack runs on the mk8s cluster and must uninstall while
# the cluster's API server and kubeconfig still exist - otherwise its
# Helm releases hang, and a load balancer one of them created can leak
# and block the network from deleting. Foreground deletion holds the
# cluster until the stack is uninstalled. Background deletion (the
# kubectl default) tears everything down at once and can orphan cloud
# resources.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}

  nodePools:
  - name: gpu-h100
    className: nebius-h100-8x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 4

  











  
    
      inference-cluster-vultr.yaml
      
          
          
        
      
    
    # An InferenceCluster backed by a Vultr Kubernetes Engine (VKE) cluster.
#
# Modelplane provisions the full VKE cluster (VPC, control plane, system + GPU
# node pools) and installs the inference stack (cert-manager, Envoy Gateway,
# Prometheus, LeaderWorkerSet, Gateway API).
#
# The system node pool that hosts control-plane components is provisioned
# automatically and is not declared here. Only GPU node pools - each
# referencing an InferenceClass that describes the hardware shape and how
# to provision it - need to be declared.
#
# The VKE kubeconfig embeds static client certificates, so Modelplane needs
# nothing beyond it to authenticate to the cluster. The Vultr API key on
# the ClusterProviderConfig named default only provisions the cluster.
#
# ModelCache RWX storage: none. VKE ships the vultr-vfs-storage
# ReadWriteMany StorageClass backed by Vultr File System, but it is not
# usable on GPU nodes - attaching a VFS-backed PVC to a GPU node fails with
# a 422 error - and Modelplane provisions no alternative. Deploy
# single-node engines without a ModelCache and let them pull weights
# directly from the model source.
#
# The system node pool (hosting control-plane components) is a fixed inline
# pool on the cluster resource. provider-vultr only honors changes to
# that inline pool at creation time - it cannot be resized after the fact.
# GPU node pools declared here are separate KubernetesNodePool resources
# with an independent lifecycle; they can be updated or removed after the
# cluster exists. Set maxNodeCount up front to enable VKE's server-side
# autoscaler.
#
# Delete this InferenceCluster with foreground cascading deletion for a
# clean teardown:
#
#   kubectl delete inferencecluster vultr-ewr --cascade=foreground
#
# The inference stack runs on the VKE cluster and must uninstall while
# the cluster's API server and kubeconfig still exist - otherwise its
# Helm releases hang, and a load balancer one of them created can leak
# and block the VPC from deleting. Foreground deletion holds the
# cluster until the stack is uninstalled. Background deletion (the
# kubectl default) tears everything down at once and can orphan cloud
# resources.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: vultr-ewr
  labels:
    modelplane.ai/region: ewr
spec:
  cluster:
    source: Vultr
    vultr:
      region: ewr

  nodePools:
    - name: gpu-l40s
      className: vultr-l40s-1x
      nodeCount: 1
      minNodeCount: 1
      maxNodeCount: 4

  











  
    
      inference-cluster-existing.yaml
      
          
          
        
      
    
    # An InferenceCluster using an existing cluster you manage yourself.
#
# Provide a kubeconfig Secret so Modelplane can install the inference
# stack and deploy models. Each GPU pool references an InferenceClass
# that describes the hardware - used by the scheduler to know what
# capacity is available.
#
# The kubeconfig Secret must exist in the control plane cluster before
# creating this InferenceCluster.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: byo-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: Existing
    existing:
      secretRef:
        name: byo-cluster-kubeconfig
        key: kubeconfig

      # Optional: a cloud identity Secret so Modelplane authenticates to the
      # cluster via the cloud's IAM instead of a token baked into the
      # kubeconfig. type selects the cloud identity and must match the cluster's
      # cloud; it defaults to GoogleApplicationCredentials. For example, a
      # Nebius managed cluster:
      # identitySecretRef:
      #   name: byo-cluster-sa-key
      #   key: credentials.json
      #   type: NebiusServiceAccountCredentials

  # Each pool's nodes must be labeled modelplane.ai/pool=<name> (here
  # modelplane.ai/pool=gpu-h100). The scheduler pins a worker to its pool by
  # this label; Modelplane provisions and labels EKS/GKE pools itself, but on a
  # BYO cluster you label the nodes. Without it worker pods stay Pending.
  nodePools:
  - name: gpu-h100
    className: h100-8x-byo
    nodeCount: 2

  







## Cache storage

A [ModelCache](/models/model-cache/) stages model weights on a
`ReadWriteMany` (RWX) StorageClass on the workload cluster. Where that comes from
depends on the source:


- **`GKE`** (Filestore Enterprise), **`EKS`** (EFS), **`AKS`** (Azure Files),
  and **`Nebius`** (shared filesystem): auto-provisioned. Those classes are
  fixed; nothing for the admin to do.
- **`Vultr`**: none. VKE's built-in RWX class (Vultr File System) isn't
  usable on GPU nodes, and Modelplane doesn't provision an alternative.
  Deploy single-node engines without a `ModelCache` and let them pull
  weights directly from the model source.
- **`Existing`**: bring your own. Create an RWX StorageClass on the cluster, with
  any backend that supports automatic PVC provisioning (WekaIO, NetApp Trident,
  `FSx` for NetApp, and similar), and name it in
  `cluster.existing.cache.storageClassName`.


The ML team's `ModelCache` and `ModelDeployment` specs are the same regardless of
which backing storage a cluster uses.



---

# Drain a Cluster

Source: /platform/drain-cluster/

**API:** [`modelplane.ai/v1alpha1` · InferenceCluster](/reference/inferenceclusters/) · [`ModelDeployment`](/reference/modeldeployments/)

Taking a cluster out of service, for maintenance or decommissioning, means
telling Modelplane to stop scheduling there and, when you can't wait for work to
finish, to move what's already running. You do this by tainting the
`InferenceCluster`. It's the fleet-level counterpart to `kubectl drain` on a
node. Modelplane reschedules the affected replicas onto other clusters whose
hardware fits, the same way it placed them to begin with.

Taints follow the Kubernetes model and come in two effects:

- **`NoSchedule`** stops new replicas landing on the cluster but leaves the ones
  already there running. Existing work finishes on its own while nothing new
  arrives.
- **`NoExecute`** also moves the replicas already there. Modelplane deletes each
  one and reschedules it onto another cluster that fits.

## Taint the cluster

Add a taint to `spec.taints`, each with a `key`, an optional `value`, and an
`effect`:

```yaml {nocopy=true}
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-us-east
spec:
  taints:
  - key: modelplane.ai/maintenance
    effect: NoSchedule    # NoExecute to also move running replicas off
  # cluster source and node pools unchanged
```

Removing the taint lets the cluster take work again. Nothing reschedules back on
its own: a taint only governs where new replicas can land, so replicas that
moved away stay where they went.

## What happens to running replicas

Under `NoExecute`, Modelplane reschedules each replica on the cluster the way it
schedules a new one, onto another cluster whose hardware satisfies the
deployment's device selectors and that isn't repelling the replica. The move
deletes the replica here and recreates it there, so the model reloads on the new
cluster and any requests still in flight to the old replica are dropped. The
deployment's other replicas keep serving while one moves.

When no other cluster can take a replica, because every candidate is full or
tainted, the deployment runs below its `spec.replicas` until capacity frees up.
Its `ReplicasScheduled` condition reports the shortfall, so a drain that can't
finish is visible rather than silent.

Under `NoSchedule`, running replicas stay put and only new placement is blocked.

## Keep a deployment through a drain

An ML team pins a critical deployment to a cluster through a drain by giving
it a matching toleration under `spec.template.spec.tolerations`. A replica that
tolerates a cluster's `NoSchedule` taint can still be placed there; one that
tolerates a `NoExecute` taint stays put when that taint is applied.

```yaml {nocopy=true}
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
spec:
  replicas: 2
  template:
    spec:
      tolerations:
      - key: modelplane.ai/maintenance
        operator: Exists    # Exists ignores value; Equal matches key and value
      # engines unchanged
```

A toleration matches a taint by `key` and `effect`. `operator: Exists` matches
any value for the key, while the default `Equal` matches key and value together;
an empty `key` with `Exists` tolerates every taint on the cluster. An empty
`effect` matches both effects. A replica is placed on, or left on, a tainted
cluster only when it tolerates every taint the cluster carries.

## Confirm the drain

Modelplane doesn't publish a per-cluster replica count. Check a drain the way
you check a drained node, by listing the replicas still placed on the cluster:

```bash
kubectl get modelreplica -l modelplane.ai/cluster=gpu-us-east
```

Once that returns nothing, or only replicas that tolerate the taint and are
meant to stay, the drain is done and you can remove the cluster.


---

# FAQ

Source: /overview/faq/



Short answers to the questions that come up first, with links to the full
treatment. If you're new here, read the [Introduction](/overview/)
and [How Modelplane works](/overview/how-it-works/) first.

## What Modelplane is


  Is Modelplane a serving engine like vLLM?
  
    No, Modelplane is the control plane <em>above the engine. It composes serving
engines like vLLM, SGLang, and NVIDIA TensorRT-LLM, and operates them across a
fleet of clusters. It doesn’t serve tokens itself. You bring the engine; Modelplane schedules
it, routes to it, scales it, and caches its weights across your inference fleet.
  




  Does Modelplane replace vLLM or SGLang?
  
    No, they run the model; Modelplane runs the fleet. A <code>ModelDeployment carries
your engine container and its flags, and Modelplane composes it onto the right
cluster. Switching or upgrading engines is a change to your deployment, not to
Modelplane.
  




  How is Modelplane different from KServe or NVIDIA Dynamo?
  
    Scope. KServe and Dynamo are cluster orchestrators: they schedule, scale, route,
and cache within a single Kubernetes cluster. Modelplane runs its operations across a
fleet of clusters, clouds, and regions. Modelplane uses llm-d for multi-node serving,
and KV-cache management, as do KServe and Dynamo. Modelplane is planning deeper integrations
with NVIDIA Dynamo in future releases.
  




  How is Modelplane different from a managed provider like Baseten or Fireworks?
  
    Managed providers run fleet-scale serving inside their own closed platform.
Modelplane is the open equivalent that runs in infrastructure you own. The
difference is open, in your own infrastructure, community-driven, and neutral
across the stack, not scope. You can still route to a managed provider from Modelplane.
  



## What it supports


  What models does Modelplane support?
  
    Modelplane supports any model, including open weights, custom models, and just about
anything that can be downloaded from Hugging Face, NVIDIA NGC, and other registries.
  




  Does Modelplane support NVIDIA?
  
    <p>Yes, across the stack. NVIDIA is the most widely available accelerator on the
clouds Modelplane runs on and the primary target today. Modelplane binds NVIDIA
GPUs to pods through Dynamic Resource Allocation (DRA), matching devices by
attributes such as GPU memory and architecture with CEL selectors.
<p>The software stack rides on the engine-agnostic API. NVIDIA NIM microservices and
the TensorRT-LLM engine run as engine containers like any other, Modelplane stages
weights and NIM-style artifacts from NVIDIA NGC alongside Hugging Face and other
registries, and the inference stack it installs includes NVIDIA Dynamo and llm-d,
with deeper Dynamo integration on the roadmap.

  




  Which engines and accelerators are supported?
  
    The API is engine-agnostic: any engine that runs as a container works, and its
flags are yours to write. Multiple accelerators are supported as long as they
can be bound through DRA, and the device model (DRA plus CEL selectors) is built to
extend to other accelerators and fabrics.
  




  Which clouds or neoclouds does Modelplane support?
  
    Today Modelplane provisions clusters on a few hyperscalers and neoclouds, and supports
bringing your own Kubernetes cluster anywhere. More provisioners are on the roadmap; the
bring-your-own path means you can run on any Kubernetes now. See
Supported Providers for the full matrix of clouds,
neoclouds, and their Crossplane providers.
  




  Can I bring my own cluster, or run on a neocloud or on-premise?
  
    Yes, an <code>InferenceCluster with <code>source: Existing registers a cluster you already
run, through its kubeconfig. Modelplane installs the serving stack it needs but
doesn’t provision the infrastructure. This is how you run on neoclouds and
on-premise today.
  



## What it requires


  Where does Modelplane run?
  
    Modelplane runs as a control plane on a control cluster: an ordinary Kubernetes
cluster with Crossplane installed, with no GPUs of its own. The inference clusters
it manages do the serving, and each needs Dynamic Resource Allocation (DRA,
Kubernetes v1.35+) to bind GPUs to pods. Modelplane assumes exclusive ownership of
every inference cluster, so dedicate each one to Modelplane rather than sharing it
with other workloads.
  





  Do I need Crossplane?
  
    Yes, Modelplane is built on Crossplane and requires it. If your
platform team already runs Crossplane to manage cloud infrastructure, Modelplane is the
same pattern applied to inference. Modelplane uses Crossplane’s function framework and shares its infrastructure providers.
  





## What it can do


  How does Modelplane decide where a model runs?
  
    Two-level matching. First it filters clusters by their labels (tier, region,
provider) against your <code>clusterSelector. Then it filters node pools by matching
your device requests, real DRA requests with CEL selectors over GPU memory,
architecture, and other attributes, against each pool’s <code>InferenceClass. It places each
replica on a cluster and pool that fits and has free capacity.
  




  Can I serve across regions and clusters behind one endpoint?
  
    Yes, that’s the point. A <code>ModelService exposes one OpenAI-compatible endpoint and
load-balances across every replica of a deployment, wherever they run.
  




  Can I route to a managed provider?
  
    Yes, a <code>ModelService can include a manually created <code>ModelEndpoint that points at
an external SaaS endpoint like Together or Baseten alongside your self-hosted
replicas, and load-balances across all of them.
  




  How do large or multi-node models work?
  
    An engine can be a gang: a leader and one or more workers that Modelplane composes
into a LeaderWorkerSet across nodes. You write the coordination (like Ray or vLLM’s data-parallel coordinator) in the engine flags, and Modelplane injects
the leader’s address so the workers can join it. Multi-node deployments stage
weights through a <code>ModelCache.
  




  What about disaggregated prefill/decode?
  
    Set <code>serving.mode: PrefillDecode and define separate prefill and decode engines.
Both run on the same cluster, hand off the KV cache over a fast fabric, and
Modelplane configures the cluster-edge routing that pairs each request. The
KV-transfer flags live in your engine config.
  




  How does scaling work?
  
    Replicas are the only scaling axis. Each replica is a complete serving instance;
scaling <code>spec.replicas adds or removes whole instances across the fleet. Because
a <code>ModelDeployment exposes the Kubernetes scale subresource, <code>kubectl scale and
KEDA work without anything extra. There’s no per-pod autoscaling inside a cluster.
  




  How are model weights handled?
  
    A <code>ModelCache stages weights once per cluster on shared (ReadWriteMany) storage,
and every pod reads them locally. Pods don’t re-download on each start, and
concurrent starts don’t race. It hydrates from Hugging Face today, is optional for
single-node deployments, and is recommended for multi-node ones.
  



## The project


  Why did you pick Modelplane as a name for the project?
  
    It’s a fusion of AI Model and Control Plane. We also like that it implies that AI models
are their own layer (or plane) in the stack.
  




  What does the logo signify?
  
    Three popsicle sticks assembled to make a model plane. Balsa wood planes were the inspiration.
  




  Is Modelplane production-ready?
  
    Modelplane is in early development and moving fast. Treat it as early software. The
platform docs are specific about what’s available today
versus what’s planned. We are building it in the open.
  




  What's the license and governance?
  
    Modelplane is Apache 2.0,
with no usage caps or token metering, and is developed in the open. It’s neutral
across models, engines, accelerators, and clouds, and is intended for donation to
a neutral open source foundation. It’s a project from Upbound, the team behind Rook
and Crossplane, both CNCF Graduated and widely adopted projects.
  




  How do I get involved?
  
    Issues, discussions, and contributions are welcome on
GitHub. See <code>CONTRIBUTING.md for
development setup and the project’s conventions.
  



## Next steps




  
    
    Get started
  
  Deploy Modelplane and serve your first model.
  →



  
    
    How Modelplane works
  
  The architecture and the control loop, in one page.
  →









---

# Laguna-S-2.1

Source: /recipes/laguna/


Poolside's Laguna-S-2.1 (118B total, 8B active MoE) served FP8 as a single
`Standalone` vLLM engine on one 8x H100 node on Nebius. The FP8 weights (~121 GiB)
fit one node with headroom for KV cache, so the engine is tensor-parallel across
the 8 GPUs over NVLink, with no gang and no prefill/decode disaggregation. Weights
stage once to a `ModelCache` on a Nebius shared filesystem and mount at `/mnt/models`.

This recipe was run end to end on Nebius (`eu-north`): serving and tool calling
validated on a single 8x H100 node. `poolside/Laguna-S-2.1-FP8` is a public
repository, so no Hugging Face token or Secret is needed. Apply the platform
side first, then the ML side.

## Validated deployments









  MoE
  118B A8B
  262,144 ctx
  vLLMSGLang


  
    Cloud
    
      
        
          


AWS
        
      
        
          


Google Cloud
        
      
        
          


Azure
        
      
        
          
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      8× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        FP8
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.25.1lmsysorg/sglang:v0.5.12.post1-cu129
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/laguna
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform


  
    
      inference-class-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceClass describing a Nebius gpu-h100-sxm node with 8x NVIDIA
# H100 80GB. Nebius sizes nodes by platform + preset rather than an
# instance type; gpu-h100-sxm + 8gpu-128vcpu-1600gb is one 8x H100 SXM
# node. The devices block describes the hardware DRA-style; it is what
# the Laguna ModelDeployment's nodeSelector matches on.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-8x
spec:
  description: "Nebius gpu-h100-sxm, 8x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 8gpu-128vcpu-1600gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 8
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 8
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # H100 80GB real usable VRAM (NVIDIA DRA driver), not nominal 80GB.
      memory: { value: "81559Mi" }

  




  
    
      inference-cluster-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # An InferenceCluster backed by a Nebius mk8s cluster with a single 8x
# H100 GPU node group. Modelplane provisions the full mk8s cluster (VPC,
# control plane, system + GPU node groups) and installs the inference
# stack. ModelCache RWX storage is auto-provisioned via a Nebius shared
# filesystem, so model-cache.yaml works unchanged.
#
# Auth is the Nebius ClusterProviderConfig named default (service-account
# key + projectID); Modelplane reuses that identity to reach the cluster.
#
# eu-north1 (Finland) keeps inference in the EU for data residency.
#
# Clean teardown - delete the ML resources first (they hold a usage on the
# cluster), then the cluster:
#   kubectl delete modeldeployment,modelservice,modelcache laguna -n ml-team
#   kubectl delete inferencecluster nebius-eu-north --cascade=foreground
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}

  nodePools:
  - name: gpu-h100
    className: nebius-h100-8x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  



## Deployment


  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The model weights, staged once per cluster on a Nebius shared filesystem
# (RWX) and mounted at /mnt/models in the serving pod, so the engine reads
# FP8 weights (~121 GiB) locally instead of pulling them from Hugging Face
# on every start.
#
# poolside/Laguna-S-2.1-FP8 is a public repo (OpenMDW-1.1), so no authSecret /
# HF token is needed. Add one only if you point this at a gated repo.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: laguna
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: poolside/Laguna-S-2.1-FP8
    sizeGiB: 200

  




  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Laguna-S-2.1 (118B total / 8B active MoE) served FP8 as a single
# Standalone vLLM engine on one 8x H100 node, weights streamed from the
# shared ModelCache at /mnt/models. FP8 weights (~121 GiB) fit one node
# with headroom for KV cache, so no gang / multi-node and no
# prefill/decode disaggregation are needed - just tensor parallelism
# across the 8 GPUs over NVLink.
#
# Notes on the engine flags:
#   --tensor-parallel-size=8 shards the model across the 8 H100s in the
#     node. FP8 is selected by the checkpoint (Laguna-S-2.1-FP8), not a flag.
#   --tool-call-parser=poolside_v1 and --reasoning-parser=poolside_v1 are
#     Poolside-specific and must exist in the engine build; --trust-remote-code
#     loads Laguna's custom modeling code from the cached repo.
#   --enable-auto-tool-choice turns on server-side tool selection.
#   --max-model-len can be lowered to shrink the KV cache footprint; 262144
#     matches Poolside's reference config.
#   The engine container must be named `engine` and listen on :8000 (Modelplane
#     scrapes 8000).
#
# H100-specific FP8 settings, carried over from a community 4x H100 build:
#   VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 selects an FP8 GEMM path that is
#     correct on Hopper; --enforce-eager avoids a CUDA-graph capture OOM on
#     H100 80GB. Validate and relax these during the run if profiling allows.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: laguna
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: laguna
      engines:
      - name: laguna
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("79Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.25.1
                command: ["vllm", "serve", "/mnt/models"]
                args:
                - --served-model-name=laguna
                - --tensor-parallel-size=8
                - --max-model-len=262144
                - --gpu-memory-utilization=0.9
                - --trust-remote-code
                - --enable-auto-tool-choice
                - --tool-call-parser=poolside_v1
                - --reasoning-parser=poolside_v1
                - --enforce-eager
                - --port=8000
                env:
                - name: VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER
                  value: "0"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms laguna -n ml-team -o jsonpath='{.status.address}'
# then call it, e.g.:
#   curl "$ADDR/v1/chat/completions" -H 'Content-Type: application/json' \
#     -d '{"model":"laguna","messages":[{"role":"user","content":"hello"}]}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: laguna
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: laguna

  



## Serving with SGLang

`model-deployment-sglang.yaml` is an alternative to the vLLM deployment above. It
serves the same model with SGLang, which has native Laguna support and the
`poolside_v1` parsers. Apply it instead of `model-deployment.yaml`.


  
    
      model-deployment-sglang.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Laguna-S-2.1 FP8 served by SGLang instead of vLLM. SGLang has native
# LagunaForCausalLM support and its own poolside_v1 reasoning and tool-call
# parsers; the reasoning parser extracts the thinking block into a separate
# reasoning field. Same single-node 8x H100 shape as the vLLM variant
# (model-deployment.yaml): tensor-parallel across the 8 GPUs, weights streamed
# from the shared ModelCache at /mnt/models.
#
# FP8 is auto-detected from the checkpoint's quantization_config, so no quant
# flag or env is needed. Reasoning extraction is opt-in: the client sends
# enable_thinking=true (chat_template_kwargs), otherwise there is no thinking
# block to split.
#
# Image lmsysorg/sglang:v0.5.12.post1-cu129 - v0.5.12 is the first release with
# Poolside Laguna support (PR #24204, which also added the poolside_v1 parsers),
# .post1 is its point bugfix, cu129 for Hopper. The image does not auto-launch a
# server, so the launch command is set explicitly; SGLang defaults to port
# 30000, so --port=8000 is required for the Modelplane engine port (Modelplane
# scrapes 8000). Poolside's trtllm_mha and DFLASH cookbook optimizations are
# left off (not validated on H100).
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: laguna
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      modelCacheRef:
        name: laguna
      engines:
      - name: laguna
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 8
              selectors:
              - cel: |
                  device.driver == "gpu.nvidia.com" && device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("79Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: lmsysorg/sglang:v0.5.12.post1-cu129
                command: ["python3", "-m", "sglang.launch_server"]
                args:
                - --model-path=/mnt/models
                - --served-model-name=laguna
                - --tp-size=8
                - --context-length=262144
                - --trust-remote-code
                - --tool-call-parser=poolside_v1
                - --reasoning-parser=poolside_v1
                - --host=0.0.0.0
                - --port=8000

  





---

# Glossary

Source: /overview/glossary/


## Modelplane

The open source control plane software. You install Modelplane on a Kubernetes
cluster (the **control cluster**). Modelplane never serves tokens itself; it
orchestrates the clusters and engines that do.

## Control cluster

The Kubernetes cluster where Modelplane runs. It needs no GPUs. It holds
Modelplane's Crossplane-based components and the API resources you apply to
declare your fleet.

## Inference cluster

A GPU cluster in the fleet where serving engines run and tokens are produced.
Modelplane can provision inference clusters on EKS, GKE, and other providers, or
you can bring your own through an `InferenceCluster` with `source: Existing`.

## Fleet

All inference clusters managed by a single Modelplane control cluster.

## Platform

The inference infrastructure the platform team
provisions using `InferenceGateway`, `InferenceClass`, and `InferenceCluster`
resources. This is distinct from Modelplane itself, which runs on the control
cluster above the fleet.

## Platform team

The infrastructure team responsible for GPU capacity. They create
`InferenceCluster`, `InferenceClass`, and `InferenceGateway` resources,
provisioning the fleet that ML teams deploy against.



## ML team



The development team deploying models. They create `ModelDeployment`,
`ModelService`, and `ModelCache` resources, declaring what a model needs without
knowing which cluster it runs on.


---

# Qwen2.5-72B

Source: /recipes/qwen2.5-72b/


A 72B dense chat model served from an AWQ INT4 quantization on a single 80 GB
GPU per replica: one `Standalone` engine fed by a `ModelCache`. The platform
side comes in two shapes - an A100 on AKS and an H100 on Nebius - and the ML
side is the same manifest for both. The deployment carries no
`clusterSelector` and two replicas, and each pool holds exactly one GPU, so
with both platforms applied one replica runs on each. The service then splits
traffic between the two GPUs by weight, which the last section uses to compare
them. To serve on just one platform, apply one tab and drop `replicas` to 1.

These manifests mirror the repository's AKS and Nebius demos. Apply the
platform side first, then the ML side.

## Validated deployments









  Dense
  72B
  8,192 ctx
  vLLM


  
    Cloud
    
      
        
          


AWS
        
      
        
          


Google Cloud
        
      
        
          
          


Azure
        
      
        
          
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        AWQ INT4
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.23.0
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/qwen2.5-72b
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform










  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  


  








  
    
      inference-class-aks.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the A100 shape on AKS. One NVIDIA A100 80GB on a
# Standard_NC24ads_A100_v4. The GPU is declared as a DRA device: the scheduler
# matches a ModelDeployment's nodeSelector against this capacity, then DRA
# binds the physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: aks-a100-1x-nc24ads
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 200
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81920Mi" }

  


  
    
      inference-cluster-aks.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # AKS InferenceCluster with one A100 node pool in swedencentral. No
# clusterSelector targets it; the ModelDeployment matches on device capacity
# alone, so it lands here or on any other compatible cluster in the fleet.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-a100-single
  labels:
    modelplane.ai/region: swedencentral
spec:
  cluster:
    source: AKS
    aks:
      location: swedencentral
  nodePools:
  - name: gpua100
    className: aks-a100-1x-nc24ads
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  











  
    
      inference-class-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the H100 shape on Nebius. One NVIDIA H100 80GB on the
# gpu-h100-sxm platform, with the cuda13.0 driver preset mk8s preinstalls on
# the pool's nodes. The GPU is declared as a DRA device: the scheduler matches
# a ModelDeployment's nodeSelector against this capacity, then DRA binds the
# physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: nebius-h100-1x-sxm
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 200
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      # The H100's usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 80GB.
      memory: { value: "81559Mi" }

  


  
    
      inference-cluster-nebius.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Nebius InferenceCluster with one H100 node pool that scales from zero: no
# minNodeCount, so the autoscaling floor defaults to nodeCount (0) and the GPU
# node only appears when a deployment's pod goes Pending.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-h100-single
  labels:
    modelplane.ai/region: eu-north
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: nebius-h100-1x-sxm
    nodeCount: 0
    maxNodeCount: 1

  







## Deployment


  
    
      model-cache.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The cache the engine serves from, hydrated once per matched cluster. The AWQ
# quant repo is public, so no authSecret is needed.
apiVersion: modelplane.ai/v1alpha1
kind: ModelCache
metadata:
  name: qwen-72b-awq
  namespace: ml-team
spec:
  source: HuggingFace
  huggingFace:
    repo: Qwen/Qwen2.5-72B-Instruct-AWQ
    sizeGiB: 150

  




  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Qwen2.5-72B Instruct (AWQ INT4) served on a single 80GB GPU per replica by
# vLLM. The model layer is cloud-agnostic: the >=70Gi CEL selector matches the
# A100 class on AKS (81920Mi) and the H100 class on Nebius (81559Mi) alike.
# Replicas are fleet-wide, not per-cluster, and each pool holds exactly one
# such GPU, so with both platforms applied the two replicas land one on the
# A100 and one on the H100. With a single platform applied, drop replicas to 1.
#
# 72B in AWQ INT4 is ~40Gi of weights, leaving room for the KV cache on an
# 80GB GPU. --max-model-len caps the context at 8192 - raise it only as far as
# the leftover VRAM allows.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  replicas: 2
  template:
    spec:
      modelCacheRef:
        name: qwen-72b-awq
      engines:
      - name: qwen-72b
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("70Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=/mnt/models
                - --served-model-name=qwen-72b
                - --max-model-len=8192

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the qwen-72b deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name,
# so this selector reaches every replica. Read the public address from
# status.address:
#   kubectl get ms qwen-72b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b

  



## Compare the A100 and the H100

Replicas are fleet-wide, not per-cluster: the deployment's `replicas: 2` means
two complete serving instances, and because each pool holds a single 80 GB GPU
they land one on the A100 and one on the H100. Modelplane labels each
replica's endpoint with the cluster it runs on, so the service can split
traffic between the platforms by weight. This service pairs the deployment
label with each cluster label and gives each GPU half of the live traffic
behind the same URL:


  
    
      model-service-split.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Splits the service's traffic between the two platforms. Each entry pairs the
# deployment label with the cluster label Modelplane puts on every endpoint, and
# the weights fix each side's share of requests regardless of how many replicas
# match. 50/50 sends both GPUs the same workload; shift the ratio to favor one.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen-72b
  namespace: ml-team
spec:
  endpoints:
  - weight: 50
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b
        modelplane.ai/cluster: aks-a100-single
  - weight: 50
    selector:
      matchLabels:
        modelplane.ai/deployment: qwen-72b
        modelplane.ai/cluster: nebius-h100-single

  



Both GPUs now serve the same workload, so their engine metrics give a direct
performance comparison: scrape each replica's latency and throughput as in
[Collecting engine metrics](/guides/collecting-engine-metrics/) and
read the two side by side. Weights are relative, so once one platform wins,
shift the 50/50 toward it - 80/20, and as far as 100/0 - without touching the
deployment.



---

# AI tools

Source: /overview/ai-tools/


The Modelplane docs are built to be read by AI assistants as well as people. You
can connect a coding agent directly to this site, pull any page as Markdown, or
point a model at a single index file that lists the whole documentation set.
Every page also carries a **Copy page** menu next to its title with the same
shortcuts.

## Connect to the MCP server

The documentation MCP server lets an assistant search these docs and read any
page in real time, so its answers track the current content instead of its
training data. It exposes two tools:

- `search_modelplane_docs`: search the docs and get back the most relevant sections with their titles, URLs, and snippets.
- `get_modelplane_doc`: fetch the full Markdown of a single page.

The server URL is:

```plaintext
https://docs.modelplane.ai/mcp
```










  
  
  
  
    Claude Code
    
    
  
  
  
  
  
    Claude Desktop
    
    
  
  
  
  
  
    Cursor
    
    
  
  
  
  
  
    VS Code
    
    
  
  
  
  
  
    Other
    
    
  
  


  









  
    bash
    
      
        
        
      
    
  
  claude mcp add --transport http modelplane-docs https://docs.modelplane.ai/mcp











Open Settings, go to Connectors, and choose <strong>Add custom connector. Name it <code>modelplane-docs, enter the server URL above, and enable the connector when you start a conversation.









<p>Open the command palette, run <strong>Cursor Settings: MCP, and add a server to <code>mcp.json:



  
    json
    
      
        
        
      
    
  
  {
  "mcpServers": {
    "modelplane-docs": {
      "url": "https://docs.modelplane.ai/mcp"
    }
  }
}









<p>Create <code>.vscode/mcp.json in your workspace:


  
    json
    
      
        
        
      
    
  
  {
  "servers": {
    "modelplane-docs": {
      "type": "http",
      "url": "https://docs.modelplane.ai/mcp"
    }
  }
}









Any MCP client that speaks the streamable HTTP transport can connect to the server URL directly. No authentication is required.





The **Copy page** menu on every page also has **Connect to Cursor** and **Connect to VS Code** shortcuts that install the server in one click.

## Read pages as Markdown

Every page is also published as raw Markdown. Add `index.md` to any page URL:

```plaintext
https://docs.modelplane.ai/models/model-deployment/index.md
```

The **Copy page** control next to each title copies that Markdown to your clipboard, and **View as Markdown** opens it in the browser. Paste it into any assistant when you want to ground a question in a specific page.

## llms.txt

For tools that index a whole site, the docs publish the [`llms.txt`](https://llmstxt.org) format:

- [`llms.txt`](/llms.txt): a short index of every page with links and descriptions.
- [`llms-full.txt`](/llms-full.txt): every page concatenated into one Markdown file.

## Page menu reference

The **Copy page** menu next to each title has these actions:

<table>
	<thead>
			<tr>
					<th>Action
					<th>What it does
			
	
	<tbody>
			<tr>
					<td>Copy page
					<td>Copies the page as Markdown to your clipboard.
			
			<tr>
					<td>View as Markdown
					<td>Opens the page as raw Markdown.
			
			<tr>
					<td>Copy MCP Server
					<td>Copies the MCP server URL to your clipboard.
			
			<tr>
					<td>Connect to Cursor
					<td>Installs the MCP server in Cursor.
			
			<tr>
					<td>Connect to VS Code
					<td>Installs the MCP server in VS Code.
			
	







---

# Architecture

Source: /architecture/


Modelplane's central design choice is to build the control plane on
[Crossplane](https://crossplane.io) rather than as a bespoke set of Kubernetes
controllers. Everything else here follows from that. This section assumes you're
comfortable with Kubernetes; the rest of the Crossplane vocabulary you need is
below.

## Crossplane in brief

[Crossplane](https://crossplane.io) extends Kubernetes to manage things beyond
the cluster, cloud infrastructure, SaaS, and in Modelplane's case inference
fleets, through the same declarative, reconciled API model. Three of its concepts
matter here:

- **Composite Resources (XRs)** are custom resources whose controller, instead of
  talking to an external API directly, declares a set of other resources that
  should exist. Every Modelplane API, `InferenceCluster`, `ModelDeployment`,
  `ModelService`, is an XR.
- **Composition functions** are that controller logic. A function is a small gRPC
  service handed the observed XR and the resources it depends on, which returns
  the desired child resources. An XR runs a pipeline of one or more functions
  every reconcile; in Modelplane each is typically a single function, so the rest
  of this section says "the function" for short.
- **Providers** are controllers that manage external systems through their own
  managed resources: `provider-gcp` and `provider-aws` for cloud APIs,
  `provider-helm` for Helm releases, `provider-kubernetes` for arbitrary objects
  on any cluster. A composition function composes these like any other resource.

Put together: a Modelplane API is an XR, its logic is a composition function, and
the function composes a mix of plain Kubernetes objects, other Modelplane XRs, and
provider resources.

The resource model mirrors Kubernetes core, one scope up:
`ModelDeployment` → `ModelReplica` → `ModelService` → `ModelEndpoint` parallels
`Deployment` → `Pod` → `Service` → `Endpoint`, but across a fleet of clusters
rather than within one. A `ModelDeployment` composes a `ModelReplica` per replica,
a `ModelReplica` composes the serving workload on its target cluster, and a
`ModelService` routes across the `ModelEndpoint`s. If you know how those core
objects relate, you already know the shape of Modelplane's.

## Why Crossplane?

Modelplane is, at its core, a system that turns declarative resources into
composed infrastructure spanning cloud accounts, many Kubernetes clusters, and
the workloads on them. That's the problem Crossplane solves, and it helps in two
ways: providers and functions.

**Providers** give us reach. Modelplane has to provision Kubernetes clusters and
all the infrastructure they need across different clouds, then install software
onto them. That's an enormous surface, and providers cover it without us rolling
our own controllers for each cloud API and Helm release.

**Functions** are where Modelplane's own logic lives, and writing it as
composition functions buys several things:

- **Business logic, not controller plumbing.** A function computes desired state
  from observed state. Crossplane handles the fiddly Kubernetes controller
  details, the watches, requeues, finalizers, and drift correction, that a
  hand-written controller gets wrong in a dozen subtle ways. Less plumbing to
  write and maintain means we move faster.
- **Testability.** A function is a pure function of its inputs, so you can test
  it as a black box: feed it an XR and its dependencies, assert on the resources
  it returns. The whole test runs in process, with no API server to stand up.
- **The right language for each job.** Functions can be written in any language.
  Modelplane's are Python, for fast iteration on the serving and scheduling logic
  and because Python is the common language of the ML world, which lowers the bar
  for contributors. The performance-sensitive distributed-systems core stays in
  Go, where Crossplane and its providers already are.

The bet underneath both is that inference infrastructure is the same shape of
problem as cloud infrastructure, which Crossplane manages well. Building on it
lets Modelplane spend its effort on the part that's actually inference-specific.

## The control cluster and the fleet

Modelplane runs on a **control cluster** and manages a fleet of **workload
clusters**, the `InferenceCluster`s. The split is deliberate: the control plane
holds no GPUs and serves no tokens. It schedules, composes, and routes; the
workload clusters do the serving.

The control cluster runs Crossplane, the Modelplane composition functions (one
per resource, each a pod Crossplane calls per reconcile), the providers, and the
control-plane gateway. It also holds every Modelplane resource and the
`ProviderConfig`s that let the providers reach each workload cluster, built from
that cluster's kubeconfig.

Crossplane core drives everything. Each reconcile it asks a function what a
resource should compose and gets back the desired resources. Core then reconciles
them, applying the provider resources that the providers act on. A function only
computes desired state. It never reaches a provider or a cluster itself.

```mermaid
flowchart TB
    subgraph control["Control cluster"]
        cp["Crossplane core"]
        fns["Modelplane functions\n(one pod per resource)"]
        prov["Providers\ngcp · aws · helm · kubernetes"]
        gw["Control-plane gateway"]
    end
    subgraph fleet["Fleet"]
        wc1["Workload cluster A"]
        wc2["Workload cluster B"]
    end
    cp <-->|"desired state (gRPC)"| fns
    cp -->|composes| prov
    cp -->|composes| gw
    prov -->|provision + install via kubeconfig| wc1
    prov -->|provision + install via kubeconfig| wc2
```

Modelplane installs a serving stack on each workload cluster: the components a
cluster needs to serve models, providing inference-aware routing through Gateway
API, multi-node serving, GPU binding through DRA, and observability, among others.
The exact components evolve, but Modelplane composes and owns all of them. For
provisioned clusters the providers also create the cluster and its node pools
first.

## How a deployment is composed

A resource composes others, which compose others, until the tree bottoms out in
provider resources and plain Kubernetes objects. A `ModelDeployment` is the
clearest example. Its function schedules the replicas, then composes a
`ModelReplica` for each, and a `ModelEndpoint` for each replica that's ready to
serve. Each `ModelReplica` function composes the serving workload, a Deployment or
a LeaderWorkerSet, onto its target workload cluster through provider-kubernetes.

```mermaid
flowchart TD
    md["ModelDeployment"]
    mr1["ModelReplica\n(cluster A)"]
    mr2["ModelReplica\n(cluster B)"]
    me1["ModelEndpoint\n(cluster A)"]
    me2["ModelEndpoint\n(cluster B)"]
    wl1["Deployment / LeaderWorkerSet\non workload cluster A"]
    wl2["Deployment / LeaderWorkerSet\non workload cluster B"]

    md --> mr1
    md --> mr2
    md --> me1
    md --> me2
    mr1 --> wl1
    mr2 --> wl2
```

The platform resources compose the same way. An `InferenceCluster` composes a
`GKECluster` or `EKSCluster` (the cloud infrastructure, via the cloud providers)
and a `ServingStack` (the per-cluster software install, via provider-helm and
provider-kubernetes). Engines bind GPUs through DRA: each `claim: DRA` device in a
member's `nodeSelector` becomes a request in the `ResourceClaim` the serving pods
claim through.

## The request path

A served request crosses two gateways, both built on Gateway API. The
**control-plane gateway** is the front door: a `ModelService` composes an
`HTTPRoute` on it that matches the service's path prefix and forwards to the
matched `ModelEndpoint`s, each of which is a `Service` pointing at a workload
cluster's gateway address. The **workload-cluster gateway** then routes from the
cluster edge to the engine pods.

```mermaid
flowchart LR
    client["Client"]
    cpgw["Control-plane gateway"]
    wcgw["Workload-cluster gateway"]
    engine["Engine pods\n(vLLM, SGLang, ...)"]

    client -->|service path| cpgw
    cpgw -->|per-replica path| wcgw
    wcgw -->|engine path| engine
```

Each hop rewrites the path: the control plane rewrites the public prefix to the
replica's path, and the workload gateway strips that down to what the engine
serves. This per-backend path rewriting is the main thing the control-plane
gateway has to support, and it narrows which Gateway API implementations can fill
the role.

Which gateway sits at each layer is internal, not part of the API. The
[`InferenceGateway`](/platform/inference-gateway/) `backend` field
is an enum precisely so the control-plane gateway can grow other options over
time. Target the `ModelService` URL rather than either gateway directly.



---

# Llama-3.1-8B

Source: /recipes/llama-3.1-8b/


An 8B dense chat model on a single NVIDIA L4. The entry recipe: one `Standalone`
engine, no cache, public weights from a Hugging Face mirror. It carries no
`clusterSelector`, so device capacity alone matches it to any compatible L4 in
the fleet.

This recipe was run end to end on GKE; the `InferenceClass`, `InferenceCluster`,
and `ModelDeployment` are the exact manifests from that run. The EKS platform
shape is the standard single-L4 recipe. It passes server validation but was not
served in this run. Apply the platform side first, then the ML side. The GKE
`InferenceCluster` carries a GCP project placeholder to edit before applying.

## Validated deployments









  Dense
  8B
  8,192 ctx
  vLLM


  
    Cloud
    
      
        
          
          


AWS
        
      
        
          
          


Google Cloud
        
      
        
          


Azure
        
      
        
          


Nebius
        
      
        
          


Vultr
        
      
    
  
  
    GPU
    
      
        <strong>A16 16G
      
        <strong>L4 24G
      
        <strong>A100 40/80G
      
        <strong>H100 80G
      
        <strong>H200 141G
      
      1× per node
    
  
  
    Serving mode
    
      
        Standalone
      
        LeaderWorker
      
        PrefillDecode
      
    
  
  
  
    Precision
    
      
        BF16
      
    
  
  
  
    Engine
    
      
        vLLM
      
        SGLang
      
        llama.cpp
      
    
  
  
  
    Image
    
      vllm/vllm-openai:v0.7.3
    
  
  
  
  
    Manifests
    
      
      
      
      docs/manifests/recipes/llama-3.1-8b
    
  

<script>
function mpShowTab(name) {
  var tabs = document.querySelectorAll('.nav-tabs .nav-link');
  var match = null;
  for (var i = 0; i 


## Platform










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  


  








  
    
      inference-class-eks.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the L4 shape on EKS, validated serving Llama-3.1-8B.
#
# One NVIDIA L4 on a g6.2xlarge. The GPU is declared as a DRA device: the
# scheduler matches a ModelDeployment's nodeSelector against this capacity, then
# DRA binds the physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: eks-l4-1x-g6
spec:
  description: "EKS g6.2xlarge, 1x NVIDIA L4"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6.2xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      # The L4's real usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 24GB.
      memory: { value: "23034Mi" }

  


  
    
      inference-cluster-eks.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # EKS InferenceCluster with one L4 node pool. No clusterSelector targets it; the
# ModelDeployment matches on device capacity alone, so it lands here or on any
# other compatible cluster in the fleet.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-l4-single
  labels:
    modelplane.ai/cloud: eks
    modelplane.ai/region: us-west
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2
  nodePools:
  - name: gpu-l4
    className: eks-l4-1x-g6
    nodeCount: 1
    zones:
    - us-west-2a
    minNodeCount: 0
    maxNodeCount: 4

  











  
    
      inference-class-gke.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # InferenceClass for the L4 shape on GKE, validated serving Llama-3.1-8B.
#
# One NVIDIA L4 on a g2-standard-8. The GPU is declared as a DRA device: the
# scheduler matches a ModelDeployment's nodeSelector against this capacity, then
# DRA binds the physical GPU to the serving pod.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-l4-1x-g2
spec:
  description: "GKE g2-standard-8, 1x NVIDIA L4"
  provisioning:
    provider: GKE
    gke:
      machineType: g2-standard-8
      diskSizeGb: 100
      accelerator:
        type: nvidia-l4
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      # The L4's real usable VRAM as the NVIDIA DRA driver reports it, not the
      # nominal 24GB.
      memory: { value: "23034Mi" }

  


  
    
      inference-cluster-gke.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # GKE InferenceCluster with one L4 node pool. Replace the project ID before
# applying. No clusterSelector targets it; the ModelDeployment matches on device
# capacity alone, so it lands here or on any other compatible cluster.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-l4-single
  labels:
    modelplane.ai/cloud: gke
    modelplane.ai/region: us-central
spec:
  cluster:
    source: GKE
    gke:
      region: us-central1
  nodePools:
  - name: gpu-l4
    className: gke-l4-1x-g2
    nodeCount: 1
    zones:
    - us-central1-a
    minNodeCount: 0
    maxNodeCount: 4

  







## Deployment


  
    
      model-deployment.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Llama-3.1-8B Instruct served on a single NVIDIA L4 by vLLM, validated end to
# end on GKE (the model layer is cloud-agnostic; the same manifest serves on EKS).
#
# 8B in bf16 is ~16Gi of weights, leaving room for the KV cache on the L4's
# ~23Gi. Llama's default context is 128K, whose KV cache does not fit beside the
# weights, so --max-model-len caps it at 8192 - raise it only as far as the
# leftover VRAM allows.
#
# Weights come from the public NousResearch mirror, so no Hugging Face token is
# needed. The gated meta-llama/Llama-3.1-8B-Instruct original needs an hf-token
# Secret on the *workload* cluster (the engine pod reads it, not the control
# plane) and HF_TOKEN passed on the engine container.
#
# No --port or --host: Modelplane's routing expects the engine on its default
# :8000 with a /health probe, and passes args through verbatim.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: llama-3-1-8b
  namespace: ml-team
spec:
  # One replica, matched to any compatible InferenceCluster by device capacity.
  replicas: 1
  template:
    spec:
      engines:
      - name: llama
        members:
    # A single self-contained vLLM pod. The container named "engine" is the
    # inference server; its image and args pass through verbatim.
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # An 8B model needs most of an L4. >=20Gi selects the L4 (which
          # reports ~23Gi) without over-constraining. DRA evaluates this CEL
          # against the InferenceClass device, then against the GPU's
          # ResourceSlice when it binds the claim.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("20Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.7.3
                args:
                - "--model=NousResearch/Meta-Llama-3.1-8B-Instruct"
            # The id clients pass as "model" in OpenAI requests.
                - "--served-model-name=llama-3.1-8b"
            # Cap the context so the KV cache fits beside the weights on the L4.
                - "--max-model-len=8192"

  




  
    
      model-service.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Exposes the llama-3-1-8b deployment's endpoints as a single OpenAI-compatible
# URL. Modelplane labels each composed ModelEndpoint with the deployment name, so
# this selector reaches every replica. Read the public address from
# status.address:
#   kubectl get ms llama-3-1-8b -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: llama-3-1-8b
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: llama-3-1-8b

  





---

# Route to External Providers

Source: /models/model-endpoint/

**API:** [`modelplane.ai/v1alpha1` · ModelEndpoint](/reference/modelendpoints/)

A `ModelEndpoint` is a single reachable inference endpoint that a
[`ModelService`](/models/model-service/) can route to. Modelplane creates
one for each of your replicas automatically, but you can also create one by hand
to point at an inference endpoint Modelplane doesn't run, most often a SaaS
provider like Together or Baseten. A service treats both the same, so you can
front your own replicas and an external provider behind one URL: send overflow to
the provider when your fleet is busy, or fail over to it as a break-glass option.

## Routing to an external provider

Create a `ModelEndpoint` with three things:


  
    
      model-endpoint.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # Modelplane composes a ModelEndpoint per ModelReplica automatically. Create one
# manually only to register an external inference endpoint with a ModelService,
# for example a SaaS provider like Together or BaseTen.
apiVersion: modelplane.ai/v1alpha1
kind: ModelEndpoint
metadata:
  name: kimi-k2-together
  namespace: ml-team
  labels:
    # 1. A label of your own for a ModelService to select on. Any label
    #    works; modelplane.ai/external-provider is a readable convention.
    modelplane.ai/external-provider: together
spec:
  # 2. The provider's base URL.
  url: https://api.together.xyz/
  # 3. The path to rewrite requests to. A ModelService receives requests at
  #    /<namespace>/<service>/v1/... and strips only the /<namespace>/<service>/
  #    prefix, so an OpenAI-compatible provider that already serves /v1/...
  #    takes just /.
  rewritePath: /

  



Then point a [`ModelService`](/models/model-service/) at it. Selecting
`modelplane.ai/external-provider: together` routes to the provider; adding a
second entry for a deployment fronts both behind one URL, so traffic can spill
over to the provider alongside your own replicas:


  
    
      model-service-external.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A ModelService's endpoints list combines: one entry can select your own
# deployment's replicas while another selects an external ModelEndpoint, so
# overflow or break-glass traffic to a SaaS provider sits behind the same URL
# as your own replicas.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: kimi-k2
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: kimi-k2          # your own replicas
  - selector:
      matchLabels:
        modelplane.ai/external-provider: together  # the endpoint above

  



The provider must speak the OpenAI API, since that's the contract a
`ModelService` exposes. Anything OpenAI-compatible works; `url` and `rewritePath`
are all that change between providers.



---

# Scale the platform

Source: /getting-started/scale-the-platform/


You have one small-GPU cluster with a running model. In this guide, you'll grow
the fleet with larger-GPU capacity so the ML team has more to schedule against.

Provisioning takes about 10 to 15 minutes.

## Register more clusters










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  


  







<p>Register two more clusters with a bigger hardware class: <code>L40S (<code>48 GB) in
<code>us-west and <code>eu-central:

  
    
      platform-scale.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: l40s-1x-g6e
spec:
  description: "EKS g6e.xlarge, 1x NVIDIA L40S"
  provisioning:
    provider: EKS
    eks:
      instanceType: g6e.xlarge
      diskSizeGb: 100
      accelerator:
        type: nvidia-l40s
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ada Lovelace }
    capacity:
      memory: { value: "46068Mi" }
---
# g6e.xlarge is available in us-east-1, us-west-2, and eu-central-1.
# eu-west-1 does NOT have g6e.xlarge.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: EKS
    eks:
      region: us-west-2
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west-2a
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: eks-eu-central
  labels:
    modelplane.ai/region: eu-central
spec:
  cluster:
    source: EKS
    eks:
      region: eu-central-1
  nodePools:
  - name: gpu-l40s
    className: l40s-1x-g6e
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - eu-central-1a

  


  
    
    Note
  
  
    <code>g6e.xlarge runs ~$2/hr on demand. Two of them plus the <code>L4 from earlier is a
few dollars for this tour. Clean up when you’re done (see Clean
up).
  









<p>Register two more clusters with a bigger hardware class: <code>A100 (<code>40 GB) in
<code>us-west and <code>us-east. Apply the manifest:

  
    
      platform-scale.yaml
      
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      # A100 40GB real reported VRAM. Keep the selector at >= 35Gi (not >= 40Gi)
      # so it reliably clears the L4 (24Gi) without hitting the boundary.
      memory: { value: "40960Mi" }
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-us-west
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-west1-b
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gpu-us-east
  labels:
    modelplane.ai/region: us-east
spec:
  cluster:
    source: GKE
    gke:
      region: us-east1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 0
    maxNodeCount: 2
    zones:
    - us-east1-b

  



  
    bash
    
      
        
        
      
    
  
  kubectl apply -f /examples/getting-started/gke/platform-scale.yaml



  
    
    Note
  
  
    <code>a2-highgpu-1g runs ~$3.50/hr on demand. Two of them plus the <code>L4 from earlier
is a few dollars for this tour. Clean up when you’re done (see Clean
up).
  









<p>Register two more clusters with a bigger hardware class: <code>A100 (<code>80 GB) in
<code>eastus and <code>southcentralus:

  
    
      platform-scale.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: a100-1x
spec:
  description: "AKS Standard_NC24ads_A100_v4, 1x NVIDIA A100 80GB"
  provisioning:
    provider: AKS
    aks:
      vmSize: Standard_NC24ads_A100_v4
      diskSizeGb: 100
      accelerator:
        type: nvidia-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
      cudaComputeCapability: { version: "8.0.0" }
    capacity:
      memory: { value: "81920Mi" }
---
# verify: Standard_NC24ads_A100_v4 quota/availability in eastus and southcentralus.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-eastus
  labels:
    modelplane.ai/region: eastus
spec:
  cluster:
    source: AKS
    aks:
      location: eastus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
---
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: aks-southcentralus
  labels:
    modelplane.ai/region: southcentralus
spec:
  cluster:
    source: AKS
    aks:
      location: southcentralus
  nodePools:
  - name: gpua100
    className: a100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  


  
    
    Note
  
  
    <code>Standard_NC24ads_A100_v4 runs ~$3.70/hr on demand. Two of them plus the <code>A10
from earlier is a few dollars for this tour. Clean up when you’re done (see Clean
up).
  









<p>Nebius projects are bound to one region, so you grow the fleet by GPU tier rather
than geography. Register a bigger <code>H100 (<code>80 GB) cluster in the same region:

  
    
      platform-scale.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: h100-1x
spec:
  description: "Nebius gpu-h100-sxm, 1x NVIDIA H100 80GB"
  provisioning:
    provider: Nebius
    nebius:
      platform: gpu-h100-sxm
      preset: 1gpu-16vcpu-200gb
      diskSizeGb: 100
      driversPreset: cuda13.0
      accelerator:
        type: nvidia-h100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Hopper }
      cudaComputeCapability: { version: "9.0.0" }
    capacity:
      memory: { value: "81559Mi" }   # H100's real reported VRAM (not the nominal 80GB)
---
# Nebius projects are bound to one region, so this cluster runs in the same
# region as the base cluster. The fleet scales by GPU tier (L40S -> H100), not
# geography. To add a region, create another project + ClusterProviderConfig.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: nebius-eu-north-h100
  labels:
    modelplane.ai/region: eu-north
    modelplane.ai/tier: h100
spec:
  cluster:
    source: Nebius
    nebius: {}
  nodePools:
  - name: gpu-h100
    className: h100-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1

  


  
    
    Note
  
  
    The <code>H100 cluster costs more per hour than the <code>L40S from earlier. Clean up
when you’re done (see Clean up).
  






Modelplane provisions the new clusters in parallel:

```bash
kubectl wait --for=condition=Ready ic --all --timeout=20m
```

## Your model keeps running

Growing the fleet doesn't disturb anything already deployed. `qwen-demo` stays
on its original cluster and the two new clusters add capacity the moment
they're `Ready` with no interruption for the ML team. A replica only moves if
its deployment changes in a way that no longer fits where it runs. 

## Next step

The fleet has grown with larger-GPU capacity. The ML team is next. [Scale the model](/getting-started/scale-the-model/) to serve it across the fleet behind a single endpoint.


---

# Supported Providers

Source: /platform/providers/

Modelplane is built on [Crossplane](https://crossplane.io) and shares its
infrastructure providers, so the set of clouds and neoclouds it reaches grows
alongside Crossplane itself. This page shows where Modelplane runs today and
where it's headed.

A provider can show up here in three ways:




  
    
    Note
  
  
    <ul>
<li><strong>Provisioning supported. Modelplane creates and manages the whole cluster
from an <code>InferenceCluster, selected through <code>provisioning.provider. GKE, EKS, AKS, Nebius mk8s, Vultr VKE work this way today.
<li><strong>Bring your own supported. Register a cluster you already run with
<code>source: Existing. This works on any provider whose Kubernetes meets
Modelplane’s requirements (Dynamic Resource Allocation and a recent Kubernetes
version), so you can run on the providers below now, ahead of native
provisioning.
<li><strong>Crossplane provider exists. A Crossplane provider is published for the
cloud. That provider is the path by which native provisioning lands, so it
marks where Modelplane can grow next.


  



## Clouds and neoclouds

Listed alphabetically, spanning hyperscalers and GPU-specialist neoclouds. Each
runs a managed Kubernetes service with GPU node pools, so the bring-your-own path
covers them all today. Where a Crossplane provider exists, it's the path to
native provisioning.

<table>
	<thead>
			<tr>
					<th>Provider / service
					<th>Accelerators
					<th>Provisioning
					<th>BYO
					<th>Crossplane
			
	
	<tbody>
			<tr>
					<td>Alibaba Cloud (ACK)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>AWS (EKS)
					<td>NVIDIA Trainium
					<td>✓
					<td>✓
					<td>
			
			<tr>
					<td>Civo (K3s)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>CoreWeave (CKS)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>none yet
			
			<tr>
					<td>Crusoe (CMK)
					<td>NVIDIA AMD
					<td>Planned
					<td>✓
					<td>none yet
			
			<tr>
					<td>DigitalOcean (DOKS)
					<td>NVIDIA AMD
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>Fluidstack
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>none yet
			
			<tr>
					<td>Google Cloud (GKE)
					<td>NVIDIA TPU
					<td>✓
					<td>✓
					<td>
			
			<tr>
					<td>Huawei Cloud (CCE)
					<td>NVIDIA Ascend
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>IBM Cloud (IKS)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>none active
			
			<tr>
					<td>Lambda
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>none yet
			
			<tr>
					<td>Linode / Akamai (LKE)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>Microsoft Azure (AKS)
					<td>NVIDIA
					<td>✓
					<td>✓
					<td>
			
			<tr>
					<td>Nebius (mk8s)
					<td>NVIDIA
					<td>✓
					<td>✓
					<td>
			
			<tr>
					<td>Oracle Cloud (OKE)
					<td>NVIDIA AMD
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>OVHcloud
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>Scaleway (Kapsule)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>Tencent Cloud (TKE)
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>
			
			<tr>
					<td>Voltage Park
					<td>NVIDIA
					<td>Planned
					<td>✓
					<td>none yet
			
			<tr>
					<td>Vultr (VKE)
					<td>NVIDIA AMD
					<td>✓
					<td>✓
					<td>
			
	







  
    
    Note
  
  
    <p><strong>On-premises and bare metal. Bring an on-prem cluster the same way as any
other: stand up Kubernetes on your own hardware (like NVIDIA DGX BasePOD
or SuperPOD) with NVIDIA Base Command Manager, Run:ai, or your own tooling, then
register it with <code>source: Existing. Provisioning it for you is on the roadmap
too. Modelplane can drive NVIDIA Base Command Manager or other bare-metal
Kubernetes provisioners through Crossplane, the same pattern it uses in the
cloud.

  


Native provisioning expands as more Crossplane providers ship; until then, the
bring-your-own path runs Modelplane on any conformant Kubernetes cluster today.




  
    
    Tip
  
  
    
<p>Don’t see your cloud or neocloud, or want to be added?
Open an issue and we’ll
track it.
  






  
    
    Register a Cluster
  
  Add a cluster to Modelplane, provisioned or bring-your-own.
  →



  
    
    Define Hardware Classes
  
  Describe the GPUs and provisioning recipe each node pool uses.
  →







---

# Recipes

Source: /recipes/

Pick a model, adjust it for your cloud and GPUs, copy the manifests, and run
them with Modelplane. Each recipe records a run that served the model
end to end on real hardware: the engine image, the GPU shape, the serving
mode, and the exact manifests.






  Browse by organization
  
  View all 9 →
  


  
    
    
    
    
    
    
      



      
        Qwen
        qwen
      
      <strong>4 recipes
    
  
    
    
    
    
    
    
      



      
        Moonshot AI
        moonshotai
      
      <strong>1 recipe
    
  
    
    
    
    
    
    
      



      
        Meta
        meta-llama
      
      <strong>1 recipe
    
  
    
    
    
    
    
    
      



      
        Z.ai
        zai-org
      
      <strong>1 recipe
    
  
    
    
    
    
    
    
      



      
        NVIDIA
        nvidia
      
      <strong>1 recipe
    
  
    
    
    
    
    
    
      



      
        Poolside
        poolside
      
      <strong>1 recipe
    
  



  Browse by cloud provider


  
    
    
    
    
    
    
      



      
        AWS
        EKS
      
      <strong>4 recipes
    
  
    
    
    
    
    
    
      



      
        Google Cloud
        GKE
      
      <strong>2 recipes
    
  
    
    
    
    
    
    
      



      
        Azure
        AKS
      
      <strong>1 recipe
    
  
    
    
    
    
    
    
      



      
        Nebius
        Nebius
      
      <strong>3 recipes
    
  
    
    
    
    
    
    
      



      
        Vultr
        Vultr
      
      <strong>1 recipe
    
  




---

# API Reference

Source: /reference/


Modelplane's API is a set of Kubernetes custom resources. Each type below has
its own page with the full specification and status schema, a runnable example, and
fields you can link to directly. For release history, see the
[GitHub releases page](https://github.com/modelplaneai/modelplane/releases).


---

# Scale the model

Source: /getting-started/scale-the-model/

A `ModelService` can front more than one `ModelDeployment`. Here you add a second
deployment, pinned to a different region, and point the same service at both. The
endpoint you already curled stays the same. Behind it, traffic now load-balances
across two regions.

```mermaid
graph LR
    subgraph fleet ["Fleet"]
        IC1["cluster-a\nsmall GPU"]
        IC2["cluster-b\nlarger GPU"]
    end

    subgraph ml ["ML team"]
        MD1["ModelDeployment\nqwen-demo"]
        MD2["ModelDeployment\nqwen-west\ntargets cluster-b"]
        MS["ModelService qwen\n/ml-team/qwen/v1/..."]
    end

    IC1 --> MD1
    IC2 --> MD2
    MD1 --> MS
    MD2 --> MS
```

## Add a second deployment

The new deployment targets a larger-GPU cluster you added in the last step. On
EKS, GKE, and AKS it pins to a second region with a `clusterSelector`. On Nebius,
which runs one region per project, the capacity selector alone routes it to the
`H100` tier:










  
  
  
  
    EKS
    
    
  
  
  
  
  
    GKE
    
    
  
  
  
  
  
    AKS
    
    
  
  
  
  
  
    Nebius
    
    
  
  


  








  
    
      model-deployment-west.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the us-west cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us-west
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # L40S (46068Mi) qualifies; L4 (23034Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("40Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment-west.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the us-west cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: us-west
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # A100 40GB (40960Mi) qualifies; L4 (23034Mi) does not.
          # Threshold at 35Gi not 40Gi to avoid the boundary on A100's exact reported VRAM.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment-west.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A second ModelDeployment, pinned to a different region with clusterSelector.
# It carries its own deployment name, so the ModelService can front it alongside
# qwen-demo and serve both from one endpoint.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  # clusterSelector filters which InferenceClusters this deployment can land on.
  # This pins the replica to the eastus cluster you added in "Scale the fleet".
  template:
    spec:
      clusterSelector:
        matchLabels:
          modelplane.ai/region: eastus
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # A100 (81920Mi) qualifies; A10 (24564Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  











  
    
      model-deployment-scale.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # A second ModelDeployment fronted alongside qwen-demo from one endpoint.
# Nebius runs one region per project, so instead of pinning to another region
# this deployment selects the larger GPU (H100) you added in "Scale the fleet".
# The VRAM selector alone routes it to the H100 cluster - no clusterSelector.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: qwen-west
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: qwen
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
          # H100 (81559Mi) qualifies; L40S (46068Mi) does not.
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("60Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: vllm/vllm-openai:v0.23.0
                args:
                - --model=Qwen/Qwen2.5-0.5B-Instruct
                - --dtype=half

  







Wait until its replica is `Ready`, then check placement. You now have one replica
per region:

```bash
kubectl get modelreplica -n ml-team
```

```shell {nocopy=true}
NAME              CLUSTER       SYNCED   READY   COMPOSITION                   AGE
qwen-demo-7323a   eks-us-east   True     True    modelreplicas.modelplane.ai   42m
qwen-west-92535   eks-us-west   True     True    modelreplicas.modelplane.ai   8m
```

## Front both with one service

Update the `ModelService` to select both deployments. Each entry in
`spec.endpoints` adds its matching replicas to the same endpoint:


  
    
      model-service-multi.yaml
      
          
            
            
            
            
            
            
            
            
            
          
          
        
          
          
        
      
    
    # The same ModelService as before, now selecting two deployments. Each entry in
# spec.endpoints adds every ModelEndpoint matching its selector to the same
# OpenAI-compatible endpoint, so /ml-team/qwen/ load-balances across qwen-demo
# and qwen-west, wherever their replicas run.
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: qwen
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-demo
  - selector:
      matchLabels:
        modelplane.ai/deployment: qwen-west

  



The endpoint URL doesn't change. Clients that had this URL before still have it;
they don't know the fleet changed. The gateway load-balances across both regions,
and losing one region keeps the other serving. Send the same request as before:

```bash
ADDRESS=$(kubectl get ms qwen -n ml-team -o jsonpath='{.status.address}')
```

```bash
kubectl run -i --rm curl-test \
  --image=curlimages/curl \
  --restart=Never \
  --env="ADDRESS=$ADDRESS" \
  -- sh -c 'curl -v "$ADDRESS/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"Qwen/Qwen2.5-0.5B-Instruct\",\"messages\":[{\"role\":\"user\",\"content\":\"What is Kubernetes in one sentence?\"}],\"max_tokens\":100}"'
```

## That's the tour

You stood up a control plane, built a multi-region GPU fleet, deployed a model
across it, and ended with one stable endpoint serving requests. The platform
team published hardware. The ML team described what the model needs. Modelplane
placed them and served behind a single endpoint.

[Clean up](/getting-started/clean-up/) tears everything down
when you're done.

For more on the resources you used:

* [InferenceClass](/platform/inference-class/)
* [InferenceCluster](/platform/inference-cluster/)
* [ModelDeployment](/models/model-deployment/)
* [ModelService](/models/model-service/)

Modelplane is in active development and we're building in the open. If you're
running your own inference fleet and want to shape where this goes, we'd love to
hear from you. Star the [repository](https://github.com/modelplaneai/modelplane),
join us in [Slack](https://slack.modelplane.ai), or read the
[manifesto](https://modelplane.ai).


---

# Clean up

Source: /getting-started/clean-up/

Delete the model resources, clusters, and finally the control plane.

## Delete model resources

Delete model resources before clusters. Deleting a cluster first leaves the
deployments reconciling against infrastructure that no longer exists.

```bash
kubectl delete md --all -n ml-team
kubectl delete ms --all -n ml-team
```

Wait for all model replicas to finish:

```bash
kubectl get modelreplica -n ml-team --watch
```

## Delete the clusters

Delete all clusters with foreground cascading deletion. The serving stack on each
workload cluster must uninstall while that cluster's API server is still
reachable. Foreground deletion holds each cluster object until its stack
finishes. Background deletion can orphan cloud resources.

```bash
kubectl delete ic --all --cascade=foreground
```

Wait until all clusters are deleted:

```bash
kubectl get ic --watch
```

## Delete the control plane

Delete the kind cluster:

```bash
kind delete cluster --name modelplane
```


---

# AKSCluster

Source: /reference/aksclusters/

An AKSCluster provisions an Azure Kubernetes Service cluster with a dedicated resource group, virtual network, and node pools for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig that consumers use to target the cluster. The kubeconfig embeds a client certificate, so consumers need no separate cloud identity.

---

# EKSCluster

Source: /reference/eksclusters/

An EKSCluster provisions an EKS cluster with dedicated node groups for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig that consumers use to target the cluster. The kubeconfig embeds a static bearer token that the AWS provider refreshes.

---

# GKECluster

Source: /reference/gkeclusters/

A GKECluster provisions a GKE cluster with dedicated node pools for GPU inference and system workloads. It outputs secrets containing the cluster kubeconfig and a GCP service account key that consumers can use to target the cluster.

---

# NebiusCluster

Source: /reference/nebiusclusters/

A NebiusCluster provisions a Nebius Managed Service for Kubernetes (mk8s) cluster with dedicated node groups for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig. The kubeconfig carries only the cluster endpoint and CA certificate - Nebius clusters authenticate every client through Nebius IAM - so consumers pair it with the credentials Secret of the config referenced by spec.credentials. The cluster is created in the project that config sets as its projectID; Nebius projects are bound to a region, so the project also determines where the cluster runs.

---

# ServingStack

Source: /reference/servingstacks/

A ServingStack installs the serving substrate (LeaderWorkerSet, Gateway API, cert-manager, Prometheus) on a Kubernetes cluster.

---

# VultrCluster

Source: /reference/vultrclusters/

A VultrCluster provisions a Vultr Kubernetes Engine (VKE) cluster with dedicated node pools for GPU inference and system workloads. It outputs a Secret containing the cluster kubeconfig. The kubeconfig embeds static client certificates, so consumers need nothing beyond it to reach the cluster. Note that provider-vultr only honors node pool changes at creation time; resize existing pools through VKE's server-side autoscaler by setting maxNodeCount up front.